[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:21 dist-test-slave-7fr8 krb5kdc[25603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:21 dist-test-slave-7fr8 krb5kdc[25603](info): set up 2 sockets
Sep 26 05:51:21 dist-test-slave-7fr8 krb5kdc[25603](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 05:51:22.150034 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:22 dist-test-slave-7fr8 krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:51:22.270272 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:41839
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:41839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:22.354873 25616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:22.355114 25616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:22.355140 25616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:22.356456 25616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:22.356519 25616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:22.356534 25616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:22.356545 25616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:22.356557 25616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:22.358253 25616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:41839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:41839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:22.358537 25616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:22.358760 25616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:22.363991 25624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:22.364055 25622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:22.364225 25616 server_base.cc:1034] running on GCE node
W20260926 05:51:22.364256 25621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:22.364506 25616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:22.364842 25616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:22.365999 25616 hybrid_clock.cc:648] HybridClock initialized: now 1790401882365945 us; error 76 us; skew 500 ppm
Sep 26 05:51:22 dist-test-slave-7fr8 krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401882, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:22.368803 25616 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:51:22.369243 25616 webserver.cc:460] Webserver started at http://127.24.255.62:45057/ using document root <none> and password file <none>
I20260926 05:51:22.369477 25616 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:22.369525 25616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:22.369653 25616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:22.370801 25616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "268f905b337941478c633d80c96950ef"
format_stamp: "Formatted at 2026-09-26 05:51:22 on dist-test-slave-7fr8"
server_key: "e470079076bdad69f8c5ece6c34371e4"
server_key_iv: "1fceb467800fafd0e6329b0dd27f513c"
server_key_version: "encryptionkey@0"
I20260926 05:51:22.371157 25616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "268f905b337941478c633d80c96950ef"
format_stamp: "Formatted at 2026-09-26 05:51:22 on dist-test-slave-7fr8"
server_key: "e470079076bdad69f8c5ece6c34371e4"
server_key_iv: "1fceb467800fafd0e6329b0dd27f513c"
server_key_version: "encryptionkey@0"
I20260926 05:51:22.372650 25616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:22.373931 25631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:22.374178 25616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:22.374272 25616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "268f905b337941478c633d80c96950ef"
format_stamp: "Formatted at 2026-09-26 05:51:22 on dist-test-slave-7fr8"
server_key: "e470079076bdad69f8c5ece6c34371e4"
server_key_iv: "1fceb467800fafd0e6329b0dd27f513c"
server_key_version: "encryptionkey@0"
I20260926 05:51:22.374352 25616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:22.400727 25616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:22.403014 25616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:22.403144 25616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:22.407181 25616 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:41839
I20260926 05:51:22.407284 25683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:41839 every 8 connection(s)
I20260926 05:51:22.407609 25616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:22.408577 25684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:22.410969 25684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef: Bootstrap starting.
I20260926 05:51:22.411538 25684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:22.411808 25684 log.cc:824] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:22.412606 25684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef: No bootstrap required, opened a new log
I20260926 05:51:22.414276 25684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268f905b337941478c633d80c96950ef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 41839 } }
I20260926 05:51:22.414431 25684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:22.414454 25684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268f905b337941478c633d80c96950ef, State: Initialized, Role: FOLLOWER
I20260926 05:51:22.414604 25684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268f905b337941478c633d80c96950ef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 41839 } }
I20260926 05:51:22.414681 25684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:22.414716 25684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:22.414767 25684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:22.415359 25684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268f905b337941478c633d80c96950ef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 41839 } }
I20260926 05:51:22.415454 25684 leader_election.cc:302] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 268f905b337941478c633d80c96950ef; no voters:
I20260926 05:51:22.415623 25684 leader_election.cc:288] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:22.415671 25687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:22.415822 25687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [term 1 LEADER]: Becoming Leader. State: Replica: 268f905b337941478c633d80c96950ef, State: Running, Role: LEADER
I20260926 05:51:22.415897 25684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:22.415994 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 25616
I20260926 05:51:22.415915 25687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268f905b337941478c633d80c96950ef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 41839 } }
I20260926 05:51:22.416095 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:22.416510 25688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "268f905b337941478c633d80c96950ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268f905b337941478c633d80c96950ef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 41839 } } }
I20260926 05:51:22.416599 25688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:22.416764 25689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 268f905b337941478c633d80c96950ef. Latest consensus state: current_term: 1 leader_uuid: "268f905b337941478c633d80c96950ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268f905b337941478c633d80c96950ef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 41839 } } }
I20260926 05:51:22.416836 25689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:22.416903 25596 external_mini_cluster.cc:1428] Setting key ce5a2dba5c978743d2efc6cce9695bce
I20260926 05:51:22.416970 25696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:22.417488 25696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:22.421046 25696 catalog_manager.cc:1347] Generated new cluster ID: 650a579be9e349c0a7aea201a203cde0
I20260926 05:51:22.421104 25696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:22.429152 25696 catalog_manager.cc:1370] Generated new certificate authority record
Sep 26 05:51:22 dist-test-slave-7fr8 krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:22.429698 25696 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:22.434468 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:22.418176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.24.255.62:41839)
0926 05:51:22.418351 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:22.418360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:22.418456 (+ 96us) server_negotiation.cc:407] Connection header received
0926 05:51:22.420181 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:22.420189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:22.420249 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:22.420406 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:22.422475 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.423911 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.425341 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.425993 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.426152 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:22.430403 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:22.430409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:22.430409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:22.430422 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:22.432271 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:22.432806 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:22.432809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:22.432810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:22.432912 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:22.433302 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:22.433303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:22.433303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:22.433576 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:22.433745 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:22.434040 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 05:51:22.434228 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260926 05:51:22.438251 25696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 268f905b337941478c633d80c96950ef: Generated new TSK 0
I20260926 05:51:22.438501 25696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:51:22.468912 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:41839
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:22.553270 25710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:22.553512 25710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:22.553543 25710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:22.555147 25710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:22.555217 25710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:22.555254 25710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:22.556967 25710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:41839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:22.557242 25710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:22.557482 25710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:22.562656 25715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:22.562664 25718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:22.562664 25716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:22.562732 25710 server_base.cc:1034] running on GCE node
I20260926 05:51:22.563051 25710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:22.563323 25710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:22.564502 25710 hybrid_clock.cc:648] HybridClock initialized: now 1790401882564473 us; error 44 us; skew 500 ppm
Sep 26 05:51:22 dist-test-slave-7fr8 krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401882, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:22.567591 25710 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:51:22.568055 25710 webserver.cc:460] Webserver started at http://127.24.255.1:36881/ using document root <none> and password file <none>
I20260926 05:51:22.568281 25710 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:22.568326 25710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:22.568452 25710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:22.569634 25710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "ee3ee37ae91445afb01510af9f41cb5c"
format_stamp: "Formatted at 2026-09-26 05:51:22 on dist-test-slave-7fr8"
server_key: "3f7f6066ac74a01d1fe8a72d51be4e8a"
server_key_iv: "a1fb55108eecf461ab52e50ddb5eb506"
server_key_version: "encryptionkey@0"
I20260926 05:51:22.570082 25710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "ee3ee37ae91445afb01510af9f41cb5c"
format_stamp: "Formatted at 2026-09-26 05:51:22 on dist-test-slave-7fr8"
server_key: "3f7f6066ac74a01d1fe8a72d51be4e8a"
server_key_iv: "a1fb55108eecf461ab52e50ddb5eb506"
server_key_version: "encryptionkey@0"
I20260926 05:51:22.571564 25710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:22.572621 25725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:22.572959 25710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:51:22.573226 25710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "ee3ee37ae91445afb01510af9f41cb5c"
format_stamp: "Formatted at 2026-09-26 05:51:22 on dist-test-slave-7fr8"
server_key: "3f7f6066ac74a01d1fe8a72d51be4e8a"
server_key_iv: "a1fb55108eecf461ab52e50ddb5eb506"
server_key_version: "encryptionkey@0"
I20260926 05:51:22.573338 25710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:22.597846 25710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:22.600469 25710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:22.600616 25710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:22.600844 25710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:22.601161 25710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:22.601195 25710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:22.601220 25710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:22.601234 25710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:22.607539 25710 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:46423
I20260926 05:51:22.607599 25838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:46423 every 8 connection(s)
I20260926 05:51:22.607951 25710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:51:22.614728 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 25710
I20260926 05:51:22.614858 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
Sep 26 05:51:22 dist-test-slave-7fr8 krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401882, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:22.615060 25596 external_mini_cluster.cc:1428] Setting key 15554a4c865e8a3735c28d077b9464a0
I20260926 05:51:22.618252 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:22.608809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:54597 (local address 127.24.255.62:41839)
0926 05:51:22.608880 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:22.608883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:22.609095 (+ 212us) server_negotiation.cc:407] Connection header received
0926 05:51:22.609604 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:22.609605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:22.609616 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:22.609669 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:22.610764 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.611484 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.612642 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.612894 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.612977 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:22.615462 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:22.615463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:22.615464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:22.615471 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:22.616870 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:22.617262 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:22.617264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:22.617264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:22.617321 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:22.617573 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:22.617574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:22.617575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:22.617699 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:22.617757 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:22.618034 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 05:51:22.618158 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:51:22.618435 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:22.608768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:41839 (local address 127.24.255.1:54597)
0926 05:51:22.608974 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:22.609014 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:22.609509 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:22.609730 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:22.609734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:22.610391 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:22.610626 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:22.610632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.611649 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.611650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:22.612513 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:22.612516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.613015 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.613017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:22.613149 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:22.613705 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:22.613714 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:22.615363 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:22.616949 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:22.616958 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:22.616961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:22.617190 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:22.617378 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:22.617379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:22.617379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:22.617523 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:22.617756 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:22.617763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:22.617911 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 05:51:22.618007 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 05:51:22.618197 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260926 05:51:22.618988 25839 heartbeater.cc:347] Connected to a master server at 127.24.255.62:41839
I20260926 05:51:22.619074 25839 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:22.619259 25839 heartbeater.cc:511] Master 127.24.255.62:41839 requested a full tablet report, sending...
I20260926 05:51:22.619931 25648 ts_manager.cc:194] Registered new tserver with Master: ee3ee37ae91445afb01510af9f41cb5c (127.24.255.1:46423)
I20260926 05:51:22.620620 25648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:54597
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:51:22.642306 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:41839
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:22.728482 25844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:22.728727 25844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:22.728754 25844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:22.730149 25844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:22.730218 25844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:22.730261 25844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:22.731930 25844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:41839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:22.732228 25844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:22.732465 25844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:22.738300 25850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:22.738300 25852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:22.738385 25849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:22.738500 25844 server_base.cc:1034] running on GCE node
I20260926 05:51:22.738708 25844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:22.738971 25844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:22.740233 25844 hybrid_clock.cc:648] HybridClock initialized: now 1790401882740206 us; error 29 us; skew 500 ppm
Sep 26 05:51:22 dist-test-slave-7fr8 krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401882, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:22.743711 25844 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:51:22.744199 25844 webserver.cc:460] Webserver started at http://127.24.255.2:40611/ using document root <none> and password file <none>
I20260926 05:51:22.744416 25844 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:22.744460 25844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:22.744553 25844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:22.745574 25844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "00c957e291c14daaa782a5a7c243e2ce"
format_stamp: "Formatted at 2026-09-26 05:51:22 on dist-test-slave-7fr8"
server_key: "3edcb9b9f05dd5581818fe785cffa4cc"
server_key_iv: "d9a1122554e4ac09ab6cd42eb1e971ae"
server_key_version: "encryptionkey@0"
I20260926 05:51:22.745882 25844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "00c957e291c14daaa782a5a7c243e2ce"
format_stamp: "Formatted at 2026-09-26 05:51:22 on dist-test-slave-7fr8"
server_key: "3edcb9b9f05dd5581818fe785cffa4cc"
server_key_iv: "d9a1122554e4ac09ab6cd42eb1e971ae"
server_key_version: "encryptionkey@0"
I20260926 05:51:22.747736 25844 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 05:51:22.748668 25859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:22.748869 25844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:22.748960 25844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "00c957e291c14daaa782a5a7c243e2ce"
format_stamp: "Formatted at 2026-09-26 05:51:22 on dist-test-slave-7fr8"
server_key: "3edcb9b9f05dd5581818fe785cffa4cc"
server_key_iv: "d9a1122554e4ac09ab6cd42eb1e971ae"
server_key_version: "encryptionkey@0"
I20260926 05:51:22.749027 25844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:22.771561 25844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:22.774559 25844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:22.774686 25844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:22.774880 25844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:22.775182 25844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:22.775216 25844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:22.775238 25844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:22.775249 25844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:22.782830 25844 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:45043
I20260926 05:51:22.782919 25972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:45043 every 8 connection(s)
I20260926 05:51:22.783203 25844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:51:22.787822 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 25844
I20260926 05:51:22.787940 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:51:22.788137 25596 external_mini_cluster.cc:1428] Setting key 14f69393da77ff723232d45276d58ee6
Sep 26 05:51:22 dist-test-slave-7fr8 krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401882, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:22.793967 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:22.784146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:48329 (local address 127.24.255.62:41839)
0926 05:51:22.784204 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:22.784206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:22.784424 (+ 218us) server_negotiation.cc:407] Connection header received
0926 05:51:22.784911 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:22.784913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:22.784929 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:22.784998 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:22.786307 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.786818 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.788177 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.788463 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.788531 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:22.790952 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:22.790954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:22.790954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:22.790963 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:22.792366 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:22.792847 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:22.792849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:22.792849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:22.792918 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:22.793219 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:22.793220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:22.793220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:22.793387 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:22.793462 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:22.793764 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 05:51:22.793861 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:51:22.794063 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:22.784073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:41839 (local address 127.24.255.2:48329)
0926 05:51:22.784250 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:22.784304 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:22.784801 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:22.785150 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:22.785155 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:22.785924 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:22.786169 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:22.786175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.786943 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.786944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:22.787943 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:22.787946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.788566 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.788567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:22.788721 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:22.789316 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:22.789325 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:22.790842 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:22.792462 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:22.792470 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:22.792472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:22.792741 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:22.793028 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:22.793029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:22.793030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:22.793145 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:22.793480 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:22.793485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:22.793630 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 05:51:22.793722 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 05:51:22.793865 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260926 05:51:22.794646 25973 heartbeater.cc:347] Connected to a master server at 127.24.255.62:41839
I20260926 05:51:22.794757 25973 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:22.795074 25973 heartbeater.cc:511] Master 127.24.255.62:41839 requested a full tablet report, sending...
I20260926 05:51:22.795491 25648 ts_manager.cc:194] Registered new tserver with Master: 00c957e291c14daaa782a5a7c243e2ce (127.24.255.2:45043)
I20260926 05:51:22.796118 25648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:48329
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:51:22.817857 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:41839
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:22.911824 25978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:22.912070 25978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:22.912096 25978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:22.913344 25978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:22.913391 25978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:22.913419 25978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:22.915097 25978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:41839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:22.915333 25978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:22.915539 25978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:22.921224 25983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:22.921231 25984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:22.921231 25986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:22.921562 25978 server_base.cc:1034] running on GCE node
I20260926 05:51:22.921763 25978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:22.922024 25978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:22.923197 25978 hybrid_clock.cc:648] HybridClock initialized: now 1790401882923182 us; error 26 us; skew 500 ppm
Sep 26 05:51:22 dist-test-slave-7fr8 krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401882, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:22.926087 25978 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:51:22.926647 25978 webserver.cc:460] Webserver started at http://127.24.255.3:42689/ using document root <none> and password file <none>
I20260926 05:51:22.926882 25978 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:22.926929 25978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:22.927053 25978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:22.928170 25978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "bc91cfb75bbf4bd1b154fd7f85e52818"
format_stamp: "Formatted at 2026-09-26 05:51:22 on dist-test-slave-7fr8"
server_key: "96a5e5cbaa3806c916bdf6932aba51b1"
server_key_iv: "690dd3aac4c48b5d27565d0c3ed75abb"
server_key_version: "encryptionkey@0"
I20260926 05:51:22.928488 25978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "bc91cfb75bbf4bd1b154fd7f85e52818"
format_stamp: "Formatted at 2026-09-26 05:51:22 on dist-test-slave-7fr8"
server_key: "96a5e5cbaa3806c916bdf6932aba51b1"
server_key_iv: "690dd3aac4c48b5d27565d0c3ed75abb"
server_key_version: "encryptionkey@0"
I20260926 05:51:22.929804 25978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 05:51:22.930600 25993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:22.930892 25978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:51:22.931109 25978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "bc91cfb75bbf4bd1b154fd7f85e52818"
format_stamp: "Formatted at 2026-09-26 05:51:22 on dist-test-slave-7fr8"
server_key: "96a5e5cbaa3806c916bdf6932aba51b1"
server_key_iv: "690dd3aac4c48b5d27565d0c3ed75abb"
server_key_version: "encryptionkey@0"
I20260926 05:51:22.931218 25978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:22.951063 25978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:22.953431 25978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:22.953562 25978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:22.953794 25978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:22.954142 25978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:22.954181 25978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:22.954206 25978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:22.954229 25978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:22.962589 25978 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:38085
I20260926 05:51:22.963037 25978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:51:22.963464 26106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:38085 every 8 connection(s)
I20260926 05:51:22.964393 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 25978
I20260926 05:51:22.964476 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:51:22.964634 25596 external_mini_cluster.cc:1428] Setting key bc8fcfe180122ce33c97dcb900907b9b
Sep 26 05:51:22 dist-test-slave-7fr8 krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401882, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:22.977007 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:22.964828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:51973 (local address 127.24.255.62:41839)
0926 05:51:22.964878 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:22.964880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:22.965451 (+ 571us) server_negotiation.cc:407] Connection header received
0926 05:51:22.966287 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:22.966288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:22.966319 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:22.966371 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:22.967698 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.968845 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.970128 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.970351 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.970425 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:22.973220 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:22.973228 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:22.973229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:22.973242 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:22.974901 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:22.975537 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:22.975539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:22.975539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:22.975621 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:22.976047 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:22.976048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:22.976048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:22.976193 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:22.976256 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:22.976763 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 05:51:22.976906 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:51:22.977314 26109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:22.965036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:41839 (local address 127.24.255.3:51973)
0926 05:51:22.965307 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:22.965360 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:22.966185 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:22.966440 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:22.966445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:22.967215 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:22.967540 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:22.967549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.968977 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.968979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:22.969971 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:22.969975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:22.970459 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:22.970461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:22.970588 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:22.971120 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:22.971129 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:22.973038 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:22.975046 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:22.975061 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:22.975065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:22.975409 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:22.975744 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:22.975746 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:22.975746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:22.975962 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:22.976362 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:22.976372 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:22.976606 (+ 234us) client_negotiation.cc:769] Sending connection context
0926 05:51:22.976729 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 05:51:22.977057 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":141,"threads_started":1}
I20260926 05:51:22.977823 26107 heartbeater.cc:347] Connected to a master server at 127.24.255.62:41839
I20260926 05:51:22.977926 26107 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:22.978116 26107 heartbeater.cc:511] Master 127.24.255.62:41839 requested a full tablet report, sending...
I20260926 05:51:22.978488 25648 ts_manager.cc:194] Registered new tserver with Master: bc91cfb75bbf4bd1b154fd7f85e52818 (127.24.255.3:38085)
I20260926 05:51:22.978866 25648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:51973
I20260926 05:51:22.988867 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 05:51:22 dist-test-slave-7fr8 krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401882, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:51:23 dist-test-slave-7fr8 krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401882, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.255.1@KRBTEST.COM
I20260926 05:51:23.015264 26114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:23.005720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.24.255.1:46423)
0926 05:51:23.006031 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:23.006036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:23.006056 (+ 20us) server_negotiation.cc:407] Connection header received
0926 05:51:23.006087 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:23.006089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:23.006136 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:23.006248 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:23.007410 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:23.008210 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:23.009079 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:23.009467 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:23.009587 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:23.012204 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:23.012216 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:23.012216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:23.012224 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:23.013913 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:23.014331 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:23.014334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:23.014335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:23.014393 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:23.014567 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:23.014568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:23.014568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:23.014797 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:23.015012 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:23.015059 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 05:51:23.015151 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":132,"threads_started":1}
W20260926 05:51:23.015791 25753 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56720
Sep 26 05:51:23 dist-test-slave-7fr8 krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:51:23 dist-test-slave-7fr8 krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.1@KRBTEST.COM
I20260926 05:51:23.043402 26114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:23.034832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.24.255.1:46423)
0926 05:51:23.034905 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:23.034907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:23.034917 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:23.034943 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:23.034943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:23.034953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:23.035056 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:23.035811 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:23.036304 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:23.037194 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:23.037502 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:23.037574 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:23.040314 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:23.040317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:23.040317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:23.040330 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:23.041908 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:23.042528 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:23.042529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:23.042530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:23.042606 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:23.042889 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:23.042890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:23.042890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:23.042996 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:23.043088 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:23.043249 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 05:51:23.043330 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:51:23.044024 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 25710
I20260926 05:51:23.051005 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 25844
I20260926 05:51:23.055783 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 25978
I20260926 05:51:23.060724 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 25616
2026-09-26T05:51:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1897 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:23 dist-test-slave-7fr8 krb5kdc[26121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:23 dist-test-slave-7fr8 krb5kdc[26121](info): set up 2 sockets
Sep 26 05:51:23 dist-test-slave-7fr8 krb5kdc[26121](info): commencing operation
krb5kdc: starting...
W20260926 05:51:24.050427 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:24 dist-test-slave-7fr8 krb5kdc[26121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:51:24.197616 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:35147
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:35147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:24.291244 26134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:24.291497 26134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:24.291524 26134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:24.292824 26134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:24.292866 26134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:24.292879 26134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:24.292891 26134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:24.292913 26134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:24.294644 26134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:35147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:35147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:24.294880 26134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:24.295071 26134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:24.301395 26140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:24.301270 26142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:24.301283 26139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:24.303369 26134 server_base.cc:1034] running on GCE node
I20260926 05:51:24.303762 26134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:24.304167 26134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:24.305456 26134 hybrid_clock.cc:648] HybridClock initialized: now 1790401884305426 us; error 35 us; skew 500 ppm
Sep 26 05:51:24 dist-test-slave-7fr8 krb5kdc[26121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401884, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:24.315546 26134 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:51:24.316679 26134 webserver.cc:460] Webserver started at http://127.24.255.62:36409/ using document root <none> and password file <none>
I20260926 05:51:24.316993 26134 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:24.317065 26134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:24.317220 26134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:24.318830 26134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "81e3f942567646fd8fb2917dd93979f9"
format_stamp: "Formatted at 2026-09-26 05:51:24 on dist-test-slave-7fr8"
server_key: "957362c47d9a4ec2647204ed3841a6c9"
server_key_iv: "aa7e8b92e95e4a1fd0c96eff94bb457a"
server_key_version: "encryptionkey@0"
I20260926 05:51:24.319259 26134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "81e3f942567646fd8fb2917dd93979f9"
format_stamp: "Formatted at 2026-09-26 05:51:24 on dist-test-slave-7fr8"
server_key: "957362c47d9a4ec2647204ed3841a6c9"
server_key_iv: "aa7e8b92e95e4a1fd0c96eff94bb457a"
server_key_version: "encryptionkey@0"
I20260926 05:51:24.320781 26134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:24.321825 26149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:24.322229 26134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:24.322404 26134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "81e3f942567646fd8fb2917dd93979f9"
format_stamp: "Formatted at 2026-09-26 05:51:24 on dist-test-slave-7fr8"
server_key: "957362c47d9a4ec2647204ed3841a6c9"
server_key_iv: "aa7e8b92e95e4a1fd0c96eff94bb457a"
server_key_version: "encryptionkey@0"
I20260926 05:51:24.322504 26134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:24.360306 26134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:24.363723 26134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:24.363896 26134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:24.368337 26134 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:35147
I20260926 05:51:24.368367 26201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:35147 every 8 connection(s)
I20260926 05:51:24.368742 26134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:24.369658 26202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:24.372768 26202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9: Bootstrap starting.
I20260926 05:51:24.373481 26202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:24.373770 26202 log.cc:824] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:24.374864 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 26134
I20260926 05:51:24.374970 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:24.375078 26202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9: No bootstrap required, opened a new log
I20260926 05:51:24.375187 25596 external_mini_cluster.cc:1428] Setting key bf5948ee57b064e84e582ec7126b8ce3
I20260926 05:51:24.376971 26202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e3f942567646fd8fb2917dd93979f9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35147 } }
I20260926 05:51:24.377135 26202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:24.377157 26202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e3f942567646fd8fb2917dd93979f9, State: Initialized, Role: FOLLOWER
I20260926 05:51:24.377281 26202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e3f942567646fd8fb2917dd93979f9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35147 } }
I20260926 05:51:24.377368 26202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:24.377406 26202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:24.377456 26202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:24.378656 26202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e3f942567646fd8fb2917dd93979f9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35147 } }
I20260926 05:51:24.378775 26202 leader_election.cc:302] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81e3f942567646fd8fb2917dd93979f9; no voters:
I20260926 05:51:24.378945 26202 leader_election.cc:288] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:24.378990 26207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:24.379106 26207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [term 1 LEADER]: Becoming Leader. State: Replica: 81e3f942567646fd8fb2917dd93979f9, State: Running, Role: LEADER
I20260926 05:51:24.379190 26207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e3f942567646fd8fb2917dd93979f9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35147 } }
I20260926 05:51:24.379220 26202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:24.379526 26208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e3f942567646fd8fb2917dd93979f9. Latest consensus state: current_term: 1 leader_uuid: "81e3f942567646fd8fb2917dd93979f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e3f942567646fd8fb2917dd93979f9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35147 } } }
I20260926 05:51:24.379602 26208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:24.379808 26211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:24.380543 26211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:24.380523 26209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81e3f942567646fd8fb2917dd93979f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e3f942567646fd8fb2917dd93979f9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35147 } } }
I20260926 05:51:24.380731 26209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:24.382313 26211 catalog_manager.cc:1347] Generated new cluster ID: 19767ad7cb01433faad42aeef8de80d4
I20260926 05:51:24.382369 26211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:51:24 dist-test-slave-7fr8 krb5kdc[26121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:24.387598 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.375815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.24.255.62:35147)
0926 05:51:24.376054 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:24.376062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:24.376113 (+ 51us) server_negotiation.cc:407] Connection header received
0926 05:51:24.376999 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:24.377008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:24.377071 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:24.377239 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:24.378084 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.379522 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.380689 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.381377 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.381482 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.383938 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:24.383943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:24.383944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:24.383957 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:24.385566 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:24.385924 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:24.385927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:24.385928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:24.386222 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:24.386436 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:24.386437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:24.386438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:24.386740 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:24.386912 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:24.387048 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 05:51:24.387209 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":113,"threads_started":1}
I20260926 05:51:24.396100 26211 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:24.396967 26211 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:24.403298 26211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81e3f942567646fd8fb2917dd93979f9: Generated new TSK 0
I20260926 05:51:24.403566 26211 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:51:24.443156 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35147
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:24.545924 26228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:24.546213 26228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:24.546245 26228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:24.547611 26228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:24.547683 26228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:24.547726 26228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:24.549479 26228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35147
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:24.549774 26228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:24.550033 26228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:51:24.557020 26228 server_base.cc:1034] running on GCE node
W20260926 05:51:24.557034 26236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:24.557400 26233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:24.557040 26234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:24.557735 26228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:24.558218 26228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:24.559506 26228 hybrid_clock.cc:648] HybridClock initialized: now 1790401884559479 us; error 43 us; skew 500 ppm
Sep 26 05:51:24 dist-test-slave-7fr8 krb5kdc[26121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401884, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:24.562405 26228 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:51:24.562963 26228 webserver.cc:460] Webserver started at http://127.24.255.1:33351/ using document root <none> and password file <none>
I20260926 05:51:24.563218 26228 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:24.563274 26228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:24.563400 26228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:24.564549 26228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "7b591b12f9a14e17bbb878445ef6d7a7"
format_stamp: "Formatted at 2026-09-26 05:51:24 on dist-test-slave-7fr8"
server_key: "fd676c00cdcb73e01bc0bbecea0a4dad"
server_key_iv: "228fd585fac4272e8fd7be67e50139a7"
server_key_version: "encryptionkey@0"
I20260926 05:51:24.565024 26228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "7b591b12f9a14e17bbb878445ef6d7a7"
format_stamp: "Formatted at 2026-09-26 05:51:24 on dist-test-slave-7fr8"
server_key: "fd676c00cdcb73e01bc0bbecea0a4dad"
server_key_iv: "228fd585fac4272e8fd7be67e50139a7"
server_key_version: "encryptionkey@0"
I20260926 05:51:24.567090 26228 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 05:51:24.568127 26243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:24.568531 26228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:24.568648 26228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "7b591b12f9a14e17bbb878445ef6d7a7"
format_stamp: "Formatted at 2026-09-26 05:51:24 on dist-test-slave-7fr8"
server_key: "fd676c00cdcb73e01bc0bbecea0a4dad"
server_key_iv: "228fd585fac4272e8fd7be67e50139a7"
server_key_version: "encryptionkey@0"
I20260926 05:51:24.568723 26228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:24.586898 26228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:24.590828 26228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:24.590996 26228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:24.591259 26228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:24.591653 26228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:24.591697 26228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:24.591804 26228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:24.591836 26228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:24.599153 26228 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:34187
I20260926 05:51:24.599197 26356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:34187 every 8 connection(s)
I20260926 05:51:24.599556 26228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:51:24.601245 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 26228
I20260926 05:51:24.601356 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:51:24.601517 25596 external_mini_cluster.cc:1428] Setting key d74d462ae7e159ca31ea91c6c0206787
Sep 26 05:51:24 dist-test-slave-7fr8 krb5kdc[26121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401884, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
I20260926 05:51:24.612448 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.600442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:38833 (local address 127.24.255.62:35147)
0926 05:51:24.600549 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:24.600551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:24.600823 (+ 272us) server_negotiation.cc:407] Connection header received
0926 05:51:24.601919 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:24.601920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:24.601932 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:24.601976 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:24.603004 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.603807 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.605356 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.605662 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.605774 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.608841 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:24.608843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:24.608843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:24.608853 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:24.610368 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:24.610869 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:24.610870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:24.610870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:24.610944 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:24.611541 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:24.611543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:24.611543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:24.611684 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:24.611755 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:24.612277 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 05:51:24.612369 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 05:51:24.612494 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.600355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35147 (local address 127.24.255.1:38833)
0926 05:51:24.600634 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:24.600674 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:24.601819 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:24.602054 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:24.602059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:24.602510 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:24.602811 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:24.602818 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.603945 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.603946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:24.605187 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:24.605191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.605806 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.605808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:24.605993 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.606692 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:24.606704 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:24.608680 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:24.610466 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:24.610477 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:24.610479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:24.610739 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:24.611148 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:24.611150 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:24.611151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:24.611413 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:24.611808 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:24.611818 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:24.611989 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 05:51:24.612114 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 05:51:24.612288 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":87,"threads_started":1}
I20260926 05:51:24.613273 26357 heartbeater.cc:347] Connected to a master server at 127.24.255.62:35147
I20260926 05:51:24.613446 26357 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:24.613641 26357 heartbeater.cc:511] Master 127.24.255.62:35147 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
I20260926 05:51:24.614853 26166 ts_manager.cc:194] Registered new tserver with Master: 7b591b12f9a14e17bbb878445ef6d7a7 (127.24.255.1:34187)
I20260926 05:51:24.615671 26166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:38833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:51:24.635377 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35147
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:24.727185 26362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:24.727409 26362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:24.727437 26362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:24.728696 26362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:24.728745 26362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:24.728773 26362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:24.730458 26362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35147
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:24.730703 26362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:24.730916 26362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:24.736567 26368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:24.736580 26370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:24.736584 26367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:24.737025 26362 server_base.cc:1034] running on GCE node
I20260926 05:51:24.737268 26362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:24.737548 26362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:24.738866 26362 hybrid_clock.cc:648] HybridClock initialized: now 1790401884738842 us; error 49 us; skew 500 ppm
Sep 26 05:51:24 dist-test-slave-7fr8 krb5kdc[26121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401884, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:24.742614 26362 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:51:24.743201 26362 webserver.cc:460] Webserver started at http://127.24.255.2:33723/ using document root <none> and password file <none>
I20260926 05:51:24.743428 26362 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:24.743480 26362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:24.743607 26362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:24.744683 26362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e"
format_stamp: "Formatted at 2026-09-26 05:51:24 on dist-test-slave-7fr8"
server_key: "76aac015a318e85aba49fbd1abe7f181"
server_key_iv: "6eeafd03cb763205dc170c050f8b3412"
server_key_version: "encryptionkey@0"
I20260926 05:51:24.745038 26362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e"
format_stamp: "Formatted at 2026-09-26 05:51:24 on dist-test-slave-7fr8"
server_key: "76aac015a318e85aba49fbd1abe7f181"
server_key_iv: "6eeafd03cb763205dc170c050f8b3412"
server_key_version: "encryptionkey@0"
I20260926 05:51:24.746497 26362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:24.747402 26377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:24.747665 26362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:24.747763 26362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e"
format_stamp: "Formatted at 2026-09-26 05:51:24 on dist-test-slave-7fr8"
server_key: "76aac015a318e85aba49fbd1abe7f181"
server_key_iv: "6eeafd03cb763205dc170c050f8b3412"
server_key_version: "encryptionkey@0"
I20260926 05:51:24.747843 26362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:24.762856 26362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:24.765656 26362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:24.765837 26362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:24.766146 26362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:24.766533 26362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:24.766570 26362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:24.766603 26362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:24.766620 26362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:24.772735 26362 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:44555
I20260926 05:51:24.772862 26490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:44555 every 8 connection(s)
I20260926 05:51:24.773161 26362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:51:24 dist-test-slave-7fr8 krb5kdc[26121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401884, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:24.781464 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 26362
I20260926 05:51:24.781550 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:51:24.781710 25596 external_mini_cluster.cc:1428] Setting key 5c80ea3f8932c2709063d1fb81cddbab
I20260926 05:51:24.783586 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.774023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:49181 (local address 127.24.255.62:35147)
0926 05:51:24.774116 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:24.774119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:24.774275 (+ 156us) server_negotiation.cc:407] Connection header received
0926 05:51:24.775316 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:24.775318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:24.775333 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:24.775385 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:24.776081 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.776593 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.777645 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.777837 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.777893 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.780318 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:24.780320 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:24.780320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:24.780329 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:24.781654 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:24.782188 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:24.782190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:24.782190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:24.782258 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:24.782620 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:24.782622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:24.782623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:24.782804 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:24.782862 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:24.783350 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 05:51:24.783492 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:51:24.783668 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.773934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35147 (local address 127.24.255.2:49181)
0926 05:51:24.774151 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:24.774193 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:24.775182 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:24.775473 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:24.775478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:24.775768 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:24.775986 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:24.775990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.776688 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.776689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:24.777550 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:24.777552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.777917 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.777918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:24.778050 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.778635 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:24.778645 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:24.780193 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:24.781744 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:24.781755 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:24.781758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:24.782083 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:24.782331 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:24.782332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:24.782333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:24.782480 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:24.782858 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:24.782865 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:24.783097 (+ 232us) client_negotiation.cc:769] Sending connection context
0926 05:51:24.783210 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 05:51:24.783403 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":86,"threads_started":1}
I20260926 05:51:24.784643 26491 heartbeater.cc:347] Connected to a master server at 127.24.255.62:35147
I20260926 05:51:24.784755 26491 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:24.784967 26491 heartbeater.cc:511] Master 127.24.255.62:35147 requested a full tablet report, sending...
I20260926 05:51:24.785521 26166 ts_manager.cc:194] Registered new tserver with Master: 86f47c59674a488e8c4b5dfd4a3c5b8e (127.24.255.2:44555)
I20260926 05:51:24.786010 26166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:49181
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:51:24.814788 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35147
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:24.905463 26496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:24.905804 26496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:24.905851 26496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:24.908133 26496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:24.908232 26496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:24.908290 26496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:24.910214 26496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35147
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:24.910516 26496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:24.910779 26496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:24.916150 26502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:24.916150 26501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:24.916414 26504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:24.916268 26496 server_base.cc:1034] running on GCE node
I20260926 05:51:24.916709 26496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:24.916981 26496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:24.918185 26496 hybrid_clock.cc:648] HybridClock initialized: now 1790401884918146 us; error 65 us; skew 500 ppm
Sep 26 05:51:24 dist-test-slave-7fr8 krb5kdc[26121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401884, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:24.920766 26496 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:51:24.921229 26496 webserver.cc:460] Webserver started at http://127.24.255.3:42145/ using document root <none> and password file <none>
I20260926 05:51:24.921443 26496 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:24.921489 26496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:24.921618 26496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:24.922719 26496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "50b1bd57b3aa4270886f0eb5216961a2"
format_stamp: "Formatted at 2026-09-26 05:51:24 on dist-test-slave-7fr8"
server_key: "91951ba8cae799e8c85cecf70b6dddb7"
server_key_iv: "0f467bc4a1a5feb4cb0f2576786cc74b"
server_key_version: "encryptionkey@0"
I20260926 05:51:24.923054 26496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "50b1bd57b3aa4270886f0eb5216961a2"
format_stamp: "Formatted at 2026-09-26 05:51:24 on dist-test-slave-7fr8"
server_key: "91951ba8cae799e8c85cecf70b6dddb7"
server_key_iv: "0f467bc4a1a5feb4cb0f2576786cc74b"
server_key_version: "encryptionkey@0"
I20260926 05:51:24.924535 26496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:24.925722 26511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:24.926040 26496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:24.926183 26496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "50b1bd57b3aa4270886f0eb5216961a2"
format_stamp: "Formatted at 2026-09-26 05:51:24 on dist-test-slave-7fr8"
server_key: "91951ba8cae799e8c85cecf70b6dddb7"
server_key_iv: "0f467bc4a1a5feb4cb0f2576786cc74b"
server_key_version: "encryptionkey@0"
I20260926 05:51:24.926275 26496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:24.935002 26496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:24.937243 26496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:24.937398 26496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:24.937629 26496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:24.937937 26496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:24.937970 26496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:24.937999 26496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:24.938025 26496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:24.944211 26496 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:36761
I20260926 05:51:24.944259 26624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:36761 every 8 connection(s)
I20260926 05:51:24.944588 26496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:51:24.951401 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 26496
I20260926 05:51:24.951505 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:51:24.951655 25596 external_mini_cluster.cc:1428] Setting key bbbf3182e0cdb3c2e276c6dd2147f79d
Sep 26 05:51:24 dist-test-slave-7fr8 krb5kdc[26121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401884, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:24.956725 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.945546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:41659 (local address 127.24.255.62:35147)
0926 05:51:24.945613 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:24.945615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:24.945883 (+ 268us) server_negotiation.cc:407] Connection header received
0926 05:51:24.946883 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:24.946885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:24.946898 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:24.947008 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:24.947821 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.948388 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.949448 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.949644 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.949705 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.952309 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:24.952311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:24.952312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:24.952320 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:24.954718 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:24.955311 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:24.955313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:24.955313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:24.955390 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:24.955821 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:24.955822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:24.955822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:24.955976 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:24.956060 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:24.956551 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 05:51:24.956642 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:51:24.956849 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.945520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35147 (local address 127.24.255.3:41659)
0926 05:51:24.945681 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:24.945770 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:24.946753 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:24.947127 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:24.947132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:24.947446 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:24.947707 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:24.947712 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.948510 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.948511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:24.949355 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:24.949358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.949752 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.949753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:24.949888 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.950517 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:24.950529 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:24.952228 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:24.954857 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:24.954866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:24.954868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:24.955186 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:24.955513 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:24.955514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:24.955515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:24.955727 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:24.956113 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:24.956121 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:24.956285 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 05:51:24.956464 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 05:51:24.956629 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260926 05:51:24.957479 26625 heartbeater.cc:347] Connected to a master server at 127.24.255.62:35147
I20260926 05:51:24.957597 26625 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:24.957805 26625 heartbeater.cc:511] Master 127.24.255.62:35147 requested a full tablet report, sending...
I20260926 05:51:24.958379 26166 ts_manager.cc:194] Registered new tserver with Master: 50b1bd57b3aa4270886f0eb5216961a2 (127.24.255.3:36761)
I20260926 05:51:24.958900 26166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:41659
I20260926 05:51:24.963850 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:51:24.973894 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.965711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.24.255.62:35147)
0926 05:51:24.965768 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:24.965770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:24.965779 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:24.965806 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:24.965806 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:24.965815 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:24.965876 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:24.966588 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.967367 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.968757 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.968999 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.969060 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.970438 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:24.970440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:24.970440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:24.970447 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:24.972101 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:24.972781 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:24.972783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:24.972790 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:24.972903 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:24.973198 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:24.973199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:24.973200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:24.973442 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:24.973588 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:24.973695 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 05:51:24.973792 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:51:24.976861 26166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:51:24.978793 26166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:24.986528 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.982164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52328 (local address 127.24.255.3:36761)
0926 05:51:24.982511 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:24.982516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:24.982530 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:51:24.982631 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:24.982634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:24.982651 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:24.982752 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:24.982968 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.983797 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.985167 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.985830 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.985911 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.985976 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:24.986362 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 05:51:24.986414 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":92,"threads_started":1}
I20260926 05:51:24.986562 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.982055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:36761 (local address 127.0.0.1:52328)
0926 05:51:24.982472 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:24.982484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:24.982543 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:24.982774 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:24.982777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:24.982779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:24.982868 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:24.982871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.983910 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.983912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:24.985062 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:24.985065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.985966 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.985967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:24.986192 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.986225 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 05:51:24.986345 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 05:51:24.986436 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":137,"threads_started":1}
I20260926 05:51:24.986835 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.982049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:34187 (local address 127.0.0.1:45400)
0926 05:51:24.982255 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:24.982297 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:24.982363 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:24.983035 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:24.983036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:24.983038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:24.983086 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:24.983088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.984453 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.984454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:24.985538 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:24.985547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.986557 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.986558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:24.986648 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.986655 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 05:51:24.986688 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 05:51:24.986750 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":89,"threads_started":1}
I20260926 05:51:24.987399 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.982215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.24.255.1:34187)
0926 05:51:24.982822 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:24.982827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:24.982837 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:24.982877 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:24.982878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:24.982894 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:24.983154 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:24.983223 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.984326 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.985711 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.986443 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.987180 (+ 737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.987250 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:24.987275 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 05:51:24.987321 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":166,"threads_started":1}
I20260926 05:51:24.987572 26559 tablet_service.cc:1467] Processing CreateTablet for tablet c02860c0ad58487a844c3465ec5602da (DEFAULT_TABLE table=test-table [id=266aee8e36684e76b7f408a2c798c640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:24.987939 26559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02860c0ad58487a844c3465ec5602da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:24.988243 26291 tablet_service.cc:1467] Processing CreateTablet for tablet c02860c0ad58487a844c3465ec5602da (DEFAULT_TABLE table=test-table [id=266aee8e36684e76b7f408a2c798c640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:24.988585 26291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02860c0ad58487a844c3465ec5602da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:24.989066 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.982307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:44555 (local address 127.0.0.1:53956)
0926 05:51:24.982712 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:24.982722 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:24.982773 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:24.983108 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:24.983109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:24.983111 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:24.983163 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:24.983164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.985721 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.985722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:24.986925 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:24.986929 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.988773 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.988774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:24.988857 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.988862 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:24.988930 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 05:51:24.988992 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":79,"threads_started":1}
I20260926 05:51:24.989086 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:24.982537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.24.255.2:44555)
0926 05:51:24.982685 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:24.982688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:24.982880 (+ 192us) server_negotiation.cc:407] Connection header received
0926 05:51:24.982925 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:24.982928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:24.982945 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:24.983024 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:24.983247 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.984326 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.987816 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:24.988586 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:24.988670 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:24.988728 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:24.988953 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 05:51:24.989011 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260926 05:51:24.990593 26425 tablet_service.cc:1467] Processing CreateTablet for tablet c02860c0ad58487a844c3465ec5602da (DEFAULT_TABLE table=test-table [id=266aee8e36684e76b7f408a2c798c640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:24.990943 26425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02860c0ad58487a844c3465ec5602da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:24.991624 26642 tablet_bootstrap.cc:492] T c02860c0ad58487a844c3465ec5602da P 50b1bd57b3aa4270886f0eb5216961a2: Bootstrap starting.
I20260926 05:51:24.992303 26642 tablet_bootstrap.cc:654] T c02860c0ad58487a844c3465ec5602da P 50b1bd57b3aa4270886f0eb5216961a2: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:24.992434 26643 tablet_bootstrap.cc:492] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7: Bootstrap starting.
I20260926 05:51:24.992802 26642 log.cc:824] T c02860c0ad58487a844c3465ec5602da P 50b1bd57b3aa4270886f0eb5216961a2: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:24.993064 26643 tablet_bootstrap.cc:654] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:24.993498 26642 tablet_bootstrap.cc:492] T c02860c0ad58487a844c3465ec5602da P 50b1bd57b3aa4270886f0eb5216961a2: No bootstrap required, opened a new log
I20260926 05:51:24.993563 26642 ts_tablet_manager.cc:1397] T c02860c0ad58487a844c3465ec5602da P 50b1bd57b3aa4270886f0eb5216961a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:51:24.993593 26643 log.cc:824] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:24.994716 26643 tablet_bootstrap.cc:492] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7: No bootstrap required, opened a new log
I20260926 05:51:24.994856 26643 ts_tablet_manager.cc:1397] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:51:24.994789 26646 tablet_bootstrap.cc:492] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e: Bootstrap starting.
I20260926 05:51:24.995388 26642 raft_consensus.cc:348] T c02860c0ad58487a844c3465ec5602da P 50b1bd57b3aa4270886f0eb5216961a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 34187 } } peers { permanent_uuid: "50b1bd57b3aa4270886f0eb5216961a2" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 36761 } } peers { permanent_uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44555 } }
I20260926 05:51:24.995579 26642 raft_consensus.cc:374] T c02860c0ad58487a844c3465ec5602da P 50b1bd57b3aa4270886f0eb5216961a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:24.995620 26642 raft_consensus.cc:729] T c02860c0ad58487a844c3465ec5602da P 50b1bd57b3aa4270886f0eb5216961a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b1bd57b3aa4270886f0eb5216961a2, State: Initialized, Role: FOLLOWER
I20260926 05:51:24.995729 26642 consensus_queue.cc:260] T c02860c0ad58487a844c3465ec5602da P 50b1bd57b3aa4270886f0eb5216961a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 34187 } } peers { permanent_uuid: "50b1bd57b3aa4270886f0eb5216961a2" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 36761 } } peers { permanent_uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44555 } }
I20260926 05:51:24.995901 26646 tablet_bootstrap.cc:654] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:24.996132 26642 ts_tablet_manager.cc:1428] T c02860c0ad58487a844c3465ec5602da P 50b1bd57b3aa4270886f0eb5216961a2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:51:24.996176 26625 heartbeater.cc:503] Master 127.24.255.62:35147 was elected leader, sending a full tablet report...
I20260926 05:51:24.996527 26646 log.cc:824] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:24.997392 26646 tablet_bootstrap.cc:492] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e: No bootstrap required, opened a new log
I20260926 05:51:24.997277 26643 raft_consensus.cc:348] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 34187 } } peers { permanent_uuid: "50b1bd57b3aa4270886f0eb5216961a2" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 36761 } } peers { permanent_uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44555 } }
I20260926 05:51:24.997470 26646 ts_tablet_manager.cc:1397] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:51:24.997476 26643 raft_consensus.cc:374] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:24.997510 26643 raft_consensus.cc:729] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b591b12f9a14e17bbb878445ef6d7a7, State: Initialized, Role: FOLLOWER
I20260926 05:51:24.997635 26643 consensus_queue.cc:260] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 34187 } } peers { permanent_uuid: "50b1bd57b3aa4270886f0eb5216961a2" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 36761 } } peers { permanent_uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44555 } }
I20260926 05:51:24.997977 26357 heartbeater.cc:503] Master 127.24.255.62:35147 was elected leader, sending a full tablet report...
I20260926 05:51:24.997921 26643 ts_tablet_manager.cc:1428] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 05:51:24.999217 26646 raft_consensus.cc:348] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 34187 } } peers { permanent_uuid: "50b1bd57b3aa4270886f0eb5216961a2" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 36761 } } peers { permanent_uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44555 } }
I20260926 05:51:24.999370 26646 raft_consensus.cc:374] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:24.999392 26646 raft_consensus.cc:729] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f47c59674a488e8c4b5dfd4a3c5b8e, State: Initialized, Role: FOLLOWER
I20260926 05:51:24.999492 26646 consensus_queue.cc:260] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 34187 } } peers { permanent_uuid: "50b1bd57b3aa4270886f0eb5216961a2" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 36761 } } peers { permanent_uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44555 } }
I20260926 05:51:24.999732 26646 ts_tablet_manager.cc:1428] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 05:51:24.999763 26491 heartbeater.cc:503] Master 127.24.255.62:35147 was elected leader, sending a full tablet report...
W20260926 05:51:25.023944 26492 tablet.cc:2369] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:51:25.070415 26649 raft_consensus.cc:482] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:25.070555 26649 raft_consensus.cc:504] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 34187 } } peers { permanent_uuid: "50b1bd57b3aa4270886f0eb5216961a2" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 36761 } } peers { permanent_uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44555 } }
I20260926 05:51:25.070910 26649 leader_election.cc:288] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50b1bd57b3aa4270886f0eb5216961a2 (127.24.255.3:36761), 86f47c59674a488e8c4b5dfd4a3c5b8e (127.24.255.2:44555)
I20260926 05:51:25.074285 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:25.071253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:38151 (local address 127.24.255.3:36761)
0926 05:51:25.071304 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:25.071306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:25.071317 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:51:25.071422 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:25.071423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:25.071432 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:25.071477 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:25.071722 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.072531 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.073527 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.073922 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.073987 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:25.073997 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:25.074167 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 05:51:25.074209 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:51:25.074287 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:25.071130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:36761 (local address 127.24.255.1:38151)
0926 05:51:25.071235 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:25.071250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:25.071320 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:25.071524 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:25.071525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:25.071527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:25.071599 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:25.071601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.072618 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.072620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:25.073439 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:25.073441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.073995 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.073996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:25.074063 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:25.074066 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:51:25.074159 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 05:51:25.074207 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":59}
I20260926 05:51:25.074398 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:25.071263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:44555 (local address 127.24.255.1:47043)
0926 05:51:25.071430 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:25.071463 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:25.071517 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:25.071831 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:25.071832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:25.071834 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:25.071911 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:25.071913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.072736 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.072737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:25.073549 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:25.073552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.074146 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.074147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:25.074225 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:25.074228 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:51:25.074273 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 05:51:25.074312 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260926 05:51:25.074389 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:25.071459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:47043 (local address 127.24.255.2:44555)
0926 05:51:25.071504 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:25.071506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:25.071517 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:51:25.071664 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:25.071665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:25.071676 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:25.071735 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:25.071996 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.072653 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.073647 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.074037 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.074148 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:25.074157 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:25.074285 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 05:51:25.074323 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:51:25.075106 26445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02860c0ad58487a844c3465ec5602da" candidate_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e" is_pre_election: true
I20260926 05:51:25.075249 26445 raft_consensus.cc:2454] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b591b12f9a14e17bbb878445ef6d7a7 in term 0.
I20260926 05:51:25.075181 26579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02860c0ad58487a844c3465ec5602da" candidate_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b1bd57b3aa4270886f0eb5216961a2" is_pre_election: true
I20260926 05:51:25.075438 26579 raft_consensus.cc:2454] T c02860c0ad58487a844c3465ec5602da P 50b1bd57b3aa4270886f0eb5216961a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b591b12f9a14e17bbb878445ef6d7a7 in term 0.
I20260926 05:51:25.075634 26246 leader_election.cc:302] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b591b12f9a14e17bbb878445ef6d7a7, 86f47c59674a488e8c4b5dfd4a3c5b8e; no voters:
I20260926 05:51:25.075781 26649 raft_consensus.cc:2781] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:25.075942 26649 raft_consensus.cc:482] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:25.075974 26649 raft_consensus.cc:3037] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:25.076660 26649 raft_consensus.cc:504] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 34187 } } peers { permanent_uuid: "50b1bd57b3aa4270886f0eb5216961a2" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 36761 } } peers { permanent_uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44555 } }
I20260926 05:51:25.076805 26649 leader_election.cc:288] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [CANDIDATE]: Term 1 election: Requested vote from peers 50b1bd57b3aa4270886f0eb5216961a2 (127.24.255.3:36761), 86f47c59674a488e8c4b5dfd4a3c5b8e (127.24.255.2:44555)
I20260926 05:51:25.077066 26445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02860c0ad58487a844c3465ec5602da" candidate_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e"
I20260926 05:51:25.077034 26579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02860c0ad58487a844c3465ec5602da" candidate_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b1bd57b3aa4270886f0eb5216961a2"
I20260926 05:51:25.077145 26445 raft_consensus.cc:3037] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:25.077154 26579 raft_consensus.cc:3037] T c02860c0ad58487a844c3465ec5602da P 50b1bd57b3aa4270886f0eb5216961a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:25.078135 26445 raft_consensus.cc:2454] T c02860c0ad58487a844c3465ec5602da P 86f47c59674a488e8c4b5dfd4a3c5b8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b591b12f9a14e17bbb878445ef6d7a7 in term 1.
I20260926 05:51:25.078238 26579 raft_consensus.cc:2454] T c02860c0ad58487a844c3465ec5602da P 50b1bd57b3aa4270886f0eb5216961a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b591b12f9a14e17bbb878445ef6d7a7 in term 1.
I20260926 05:51:25.078329 26246 leader_election.cc:302] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b591b12f9a14e17bbb878445ef6d7a7, 86f47c59674a488e8c4b5dfd4a3c5b8e; no voters:
I20260926 05:51:25.078518 26649 raft_consensus.cc:2781] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:25.078697 26649 raft_consensus.cc:686] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [term 1 LEADER]: Becoming Leader. State: Replica: 7b591b12f9a14e17bbb878445ef6d7a7, State: Running, Role: LEADER
I20260926 05:51:25.078810 26649 consensus_queue.cc:237] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 34187 } } peers { permanent_uuid: "50b1bd57b3aa4270886f0eb5216961a2" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 36761 } } peers { permanent_uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44555 } }
I20260926 05:51:25.079741 26165 catalog_manager.cc:5495] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b591b12f9a14e17bbb878445ef6d7a7 (127.24.255.1). New cstate: current_term: 1 leader_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b591b12f9a14e17bbb878445ef6d7a7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 34187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50b1bd57b3aa4270886f0eb5216961a2" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 36761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86f47c59674a488e8c4b5dfd4a3c5b8e" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44555 } health_report { overall_health: UNKNOWN } } }
W20260926 05:51:25.100461 26358 tablet.cc:2369] T c02860c0ad58487a844c3465ec5602da P 7b591b12f9a14e17bbb878445ef6d7a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 05:51:25 dist-test-slave-7fr8 krb5kdc[26121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.1@KRBTEST.COM
I20260926 05:51:25.112406 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:25.104205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.24.255.1:34187)
0926 05:51:25.104258 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:25.104260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:25.104270 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:25.104333 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:25.104334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:25.104347 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:25.104407 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:25.104955 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.105441 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.106188 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.106381 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.106445 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:25.109144 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:25.109160 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:25.109160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:25.109168 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:25.110924 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:25.111377 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:25.111381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:25.111381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:25.111458 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:25.111723 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:25.111724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:25.111725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:25.111962 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:25.112126 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:25.112205 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 05:51:25.112303 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Sep 26 05:51:25 dist-test-slave-7fr8 krb5kdc[26121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:51:25 dist-test-slave-7fr8 krb5kdc[26121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.255.1@KRBTEST.COM
I20260926 05:51:25.137892 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:25.130145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45420 (local address 127.24.255.1:34187)
0926 05:51:25.130199 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:25.130201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:25.130209 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:25.130235 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:25.130236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:25.130244 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:25.130291 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:25.130982 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.131702 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.132497 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.132727 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.132784 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:25.135174 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:25.135175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:25.135176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:25.135185 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:25.136434 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:25.136986 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:25.136987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:25.136988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:25.137050 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:25.137234 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:25.137235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:25.137236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:25.137358 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:25.137456 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:25.137647 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 05:51:25.137732 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 05:51:25 dist-test-slave-7fr8 krb5kdc[26121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:51:25 dist-test-slave-7fr8 krb5kdc[26121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.1@KRBTEST.COM
I20260926 05:51:25.164001 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:25.156097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.24.255.1:34187)
0926 05:51:25.156162 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:25.156164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:25.156274 (+ 110us) server_negotiation.cc:407] Connection header received
0926 05:51:25.156307 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:25.156309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:25.156322 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:25.156395 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:25.157083 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.157627 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.158493 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:25.158691 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:25.158748 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:25.161290 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:25.161293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:25.161293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:25.161300 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:25.162739 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:25.163101 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:25.163102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:25.163102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:25.163156 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:25.163465 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:25.163466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:25.163467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:25.163594 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:25.163677 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:25.163824 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 05:51:25.163908 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 05:51:25.164486 26271 tablet_service.cc:2908] Rejecting scan request for tablet c02860c0ad58487a844c3465ec5602da: Uninitialized: clean time has not yet been initialized
I20260926 05:51:25.165313 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 26228
I20260926 05:51:25.170531 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 26362
I20260926 05:51:25.174847 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 26496
I20260926 05:51:25.178872 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 26134
2026-09-26T05:51:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2117 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:25 dist-test-slave-7fr8 krb5kdc[26666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:25 dist-test-slave-7fr8 krb5kdc[26666](info): set up 2 sockets
Sep 26 05:51:25 dist-test-slave-7fr8 krb5kdc[26666](info): commencing operation
krb5kdc: starting...
W20260926 05:51:26.166071 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:26 dist-test-slave-7fr8 krb5kdc[26666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:51:26.302898 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:38291
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:38291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:26.392670 26679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:26.393111 26679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:26.393153 26679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:26.396056 26679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:26.396167 26679 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 05:51:26.396183 26679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:26.396198 26679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:26.396210 26679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:26.396224 26679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:26.399364 26679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:38291
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:38291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:26.399675 26679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:26.399869 26679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:51:26.405434 26679 server_base.cc:1034] running on GCE node
W20260926 05:51:26.405405 26684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:26.405323 26687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:26.405323 26685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:26.405885 26679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:26.406250 26679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:26.407483 26679 hybrid_clock.cc:648] HybridClock initialized: now 1790401886407429 us; error 59 us; skew 500 ppm
Sep 26 05:51:26 dist-test-slave-7fr8 krb5kdc[26666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:26.411469 26679 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:51:26.412053 26679 webserver.cc:460] Webserver started at http://127.24.255.62:35353/ using document root <none> and password file <none>
I20260926 05:51:26.412290 26679 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:26.412344 26679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:26.412492 26679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:26.413658 26679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "547623e2a679479fbf52c9272afaa327"
format_stamp: "Formatted at 2026-09-26 05:51:26 on dist-test-slave-7fr8"
server_key: "bc8ef87f6be292f114d63cc936db59d6"
server_key_iv: "6564500823881dbba958d36989a0c160"
server_key_version: "encryptionkey@0"
I20260926 05:51:26.414027 26679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "547623e2a679479fbf52c9272afaa327"
format_stamp: "Formatted at 2026-09-26 05:51:26 on dist-test-slave-7fr8"
server_key: "bc8ef87f6be292f114d63cc936db59d6"
server_key_iv: "6564500823881dbba958d36989a0c160"
server_key_version: "encryptionkey@0"
I20260926 05:51:26.415898 26679 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260926 05:51:26.416942 26694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:26.417495 26679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:26.417614 26679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "547623e2a679479fbf52c9272afaa327"
format_stamp: "Formatted at 2026-09-26 05:51:26 on dist-test-slave-7fr8"
server_key: "bc8ef87f6be292f114d63cc936db59d6"
server_key_iv: "6564500823881dbba958d36989a0c160"
server_key_version: "encryptionkey@0"
I20260926 05:51:26.417707 26679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:26.468819 26679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:26.471668 26679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:26.471818 26679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:26.476382 26679 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:38291
I20260926 05:51:26.476634 26756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:38291 every 8 connection(s)
I20260926 05:51:26.476807 26679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:26.477557 26757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:26.480363 26757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327: Bootstrap starting.
I20260926 05:51:26.480734 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 26679
I20260926 05:51:26.480854 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:26.481051 25596 external_mini_cluster.cc:1428] Setting key 96a4d25541c8b8db3efc16e31cf173fc
I20260926 05:51:26.481570 26757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:26.482177 26757 log.cc:824] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:26.483462 26757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327: No bootstrap required, opened a new log
I20260926 05:51:26.485984 26757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547623e2a679479fbf52c9272afaa327" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38291 } }
I20260926 05:51:26.486296 26757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:26.486344 26757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547623e2a679479fbf52c9272afaa327, State: Initialized, Role: FOLLOWER
I20260926 05:51:26.486501 26757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547623e2a679479fbf52c9272afaa327" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38291 } }
I20260926 05:51:26.486586 26757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:26.486616 26757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:26.486661 26757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:26.487375 26757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547623e2a679479fbf52c9272afaa327" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38291 } }
I20260926 05:51:26.487634 26757 leader_election.cc:302] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547623e2a679479fbf52c9272afaa327; no voters:
I20260926 05:51:26.487851 26757 leader_election.cc:288] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:26.488029 26762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:26.488173 26757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 05:51:26 dist-test-slave-7fr8 krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:26.488198 26762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [term 1 LEADER]: Becoming Leader. State: Replica: 547623e2a679479fbf52c9272afaa327, State: Running, Role: LEADER
I20260926 05:51:26.489867 26762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547623e2a679479fbf52c9272afaa327" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38291 } }
I20260926 05:51:26.490767 26764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 547623e2a679479fbf52c9272afaa327. Latest consensus state: current_term: 1 leader_uuid: "547623e2a679479fbf52c9272afaa327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547623e2a679479fbf52c9272afaa327" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38291 } } }
I20260926 05:51:26.490893 26764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:26.490739 26763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "547623e2a679479fbf52c9272afaa327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547623e2a679479fbf52c9272afaa327" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38291 } } }
I20260926 05:51:26.491129 26763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:26.491377 26776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:26.491925 26776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:26.494715 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:26.481708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.24.255.62:38291)
0926 05:51:26.482067 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:26.482175 (+ 108us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:26.482216 (+ 41us) server_negotiation.cc:407] Connection header received
0926 05:51:26.483088 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:26.483101 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:26.483172 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:26.483334 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:26.484454 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.485660 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.486979 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.487558 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.487665 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:26.490369 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:26.490377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:26.490378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:26.490398 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:26.492855 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:26.493322 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:26.493326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:26.493326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:26.493453 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:26.493671 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:26.493672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:26.493672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:26.494005 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:26.494186 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:26.494308 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 05:51:26.494495 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":146,"threads_started":1}
I20260926 05:51:26.494855 26776 catalog_manager.cc:1347] Generated new cluster ID: ce3e73425de8486aa30b39ea1bc7f511
I20260926 05:51:26.494887 26776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:26.503839 26776 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:26.504668 26776 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:26.510414 26776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327: Generated new TSK 0
I20260926 05:51:26.510695 26776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:51:26.543110 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38291
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:26.634661 26783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:26.634910 26783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:26.634939 26783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:26.636399 26783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:26.636468 26783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:26.636492 26783 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:51:26.636510 26783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:26.638254 26783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.255.62:38291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:26.638523 26783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:26.638767 26783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:26.644699 26791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:26.644707 26788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:26.644699 26789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:26.645030 26783 server_base.cc:1034] running on GCE node
I20260926 05:51:26.645251 26783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:26.645529 26783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:26.646703 26783 hybrid_clock.cc:648] HybridClock initialized: now 1790401886646648 us; error 71 us; skew 500 ppm
Sep 26 05:51:26 dist-test-slave-7fr8 krb5kdc[26666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:26.649672 26783 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:51:26.650239 26783 webserver.cc:460] Webserver started at http://127.24.255.1:37105/ using document root <none> and password file <none>
I20260926 05:51:26.650470 26783 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:26.650520 26783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:26.650652 26783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:26.651770 26783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "067e6de0cd3d4506b77e97217e56c927"
format_stamp: "Formatted at 2026-09-26 05:51:26 on dist-test-slave-7fr8"
server_key: "bea05b2e72937ba6bb5325066c894757"
server_key_iv: "63cb3a0753f1d09dbc21e378b2febf86"
server_key_version: "encryptionkey@0"
I20260926 05:51:26.652136 26783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "067e6de0cd3d4506b77e97217e56c927"
format_stamp: "Formatted at 2026-09-26 05:51:26 on dist-test-slave-7fr8"
server_key: "bea05b2e72937ba6bb5325066c894757"
server_key_iv: "63cb3a0753f1d09dbc21e378b2febf86"
server_key_version: "encryptionkey@0"
I20260926 05:51:26.654311 26783 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 05:51:26.655272 26798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:26.655509 26783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:26.655609 26783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "067e6de0cd3d4506b77e97217e56c927"
format_stamp: "Formatted at 2026-09-26 05:51:26 on dist-test-slave-7fr8"
server_key: "bea05b2e72937ba6bb5325066c894757"
server_key_iv: "63cb3a0753f1d09dbc21e378b2febf86"
server_key_version: "encryptionkey@0"
I20260926 05:51:26.655704 26783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:26.676110 26783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:26.679523 26783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:26.679906 26783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:26.680598 26783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:26.680644 26783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:26.680675 26783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:26.680691 26783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:51:26 dist-test-slave-7fr8 krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:26.690446 26783 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:41231
I20260926 05:51:26.690568 26913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:41231 every 8 connection(s)
I20260926 05:51:26.690847 26783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:51:26.693889 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:26.681117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38291 (local address 127.24.255.1:60559)
0926 05:51:26.681292 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:26.681331 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:26.681865 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:26.682202 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:26.682207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:26.682514 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:26.682726 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:26.682730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.683442 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.683443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:26.684403 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:26.684405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.685110 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.685112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:26.685295 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:26.686736 (+ 1441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:26.686754 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:26.690472 (+ 3718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:26.692222 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:26.692230 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:26.692232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:26.692501 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:26.692835 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:26.692836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:26.692836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:26.692959 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:26.693269 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:26.693275 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:26.693412 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 05:51:26.693562 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 05:51:26.693710 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260926 05:51:26.694041 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:26.681140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:60559 (local address 127.24.255.62:38291)
0926 05:51:26.681189 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:26.681190 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:26.681431 (+ 241us) server_negotiation.cc:407] Connection header received
0926 05:51:26.682008 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:26.682009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:26.682022 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:26.682118 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:26.682830 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.683362 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.684522 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.684769 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.684843 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:26.690626 (+ 5783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:26.690628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:26.690629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:26.690640 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:26.692118 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:26.692619 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:26.692621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:26.692621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:26.692685 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:26.693062 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:26.693064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:26.693064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:26.693198 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:26.693263 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:26.693764 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 05:51:26.693911 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:51:26.694892 26914 heartbeater.cc:347] Connected to a master server at 127.24.255.62:38291
I20260926 05:51:26.695019 26914 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:26.695222 26914 heartbeater.cc:511] Master 127.24.255.62:38291 requested a full tablet report, sending...
I20260926 05:51:26.696071 26710 ts_manager.cc:194] Registered new tserver with Master: 067e6de0cd3d4506b77e97217e56c927 (127.24.255.1:41231)
I20260926 05:51:26.696750 26710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:60559
I20260926 05:51:26.698827 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 26783
I20260926 05:51:26.699014 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:51:26.699229 25596 external_mini_cluster.cc:1428] Setting key 948a710458b9518c91790f2c46a36d7d
I20260926 05:51:26.703280 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:26.696183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:44655 (local address 127.24.255.62:38291)
0926 05:51:26.696230 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:26.696232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:26.697068 (+ 836us) server_negotiation.cc:407] Connection header received
0926 05:51:26.697127 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:26.697128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:26.697139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:26.697193 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:26.697540 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.698031 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.698876 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.699132 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.699196 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:26.700651 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:26.700652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:26.700653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:26.700660 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:26.702191 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:26.702544 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:26.702545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:26.702545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:26.702600 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:26.702762 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:26.702763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:26.702763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:26.702882 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:26.702939 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:26.703084 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 05:51:26.703172 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:51:26.703290 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:26.696112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38291 (local address 127.24.255.1:44655)
0926 05:51:26.696985 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:26.696998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:26.697068 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:26.697205 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:26.697206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:26.697404 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:26.697469 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:26.697471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.698156 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.698157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:26.698752 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:26.698755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.699198 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.699200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:26.699282 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:26.699971 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:26.699977 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:26.700511 (+ 534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:26.702287 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:26.702288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:26.702289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:26.702476 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:26.702655 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:26.702656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:26.702656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:26.702712 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:26.702938 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:26.702939 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:26.702994 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 05:51:26.703052 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 05:51:26.703176 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:51:26.727612 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38291
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:26.814888 26923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:26.815133 26923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:26.815171 26923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:26.816485 26923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:26.816545 26923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:26.816576 26923 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:51:26.816602 26923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:26.818225 26923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.255.62:38291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:26.818506 26923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:26.818854 26923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:26.824383 26931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:26.824550 26923 server_base.cc:1034] running on GCE node
W20260926 05:51:26.824409 26928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:26.824373 26929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:26.824903 26923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:26.825167 26923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:26.826333 26923 hybrid_clock.cc:648] HybridClock initialized: now 1790401886826288 us; error 58 us; skew 500 ppm
Sep 26 05:51:26 dist-test-slave-7fr8 krb5kdc[26666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:26.829087 26923 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:51:26.829523 26923 webserver.cc:460] Webserver started at http://127.24.255.2:45977/ using document root <none> and password file <none>
I20260926 05:51:26.829730 26923 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:26.829775 26923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:26.829895 26923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:26.831027 26923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "66f2951f40f64f098ca4b65e608159f6"
format_stamp: "Formatted at 2026-09-26 05:51:26 on dist-test-slave-7fr8"
server_key: "541aecb10108941bf42a9d15de64610d"
server_key_iv: "1211a1e2d3b6021aea02571631c76062"
server_key_version: "encryptionkey@0"
I20260926 05:51:26.831403 26923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "66f2951f40f64f098ca4b65e608159f6"
format_stamp: "Formatted at 2026-09-26 05:51:26 on dist-test-slave-7fr8"
server_key: "541aecb10108941bf42a9d15de64610d"
server_key_iv: "1211a1e2d3b6021aea02571631c76062"
server_key_version: "encryptionkey@0"
I20260926 05:51:26.833297 26923 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 05:51:26.834292 26938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:26.834512 26923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:26.834605 26923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "66f2951f40f64f098ca4b65e608159f6"
format_stamp: "Formatted at 2026-09-26 05:51:26 on dist-test-slave-7fr8"
server_key: "541aecb10108941bf42a9d15de64610d"
server_key_iv: "1211a1e2d3b6021aea02571631c76062"
server_key_version: "encryptionkey@0"
I20260926 05:51:26.834689 26923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:26.843657 26923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:26.846390 26923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:26.846558 26923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:26.847281 26923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:26.847334 26923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:26.847362 26923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:26.847378 26923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:26.853924 26923 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:43625
I20260926 05:51:26.854064 27053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:43625 every 8 connection(s)
I20260926 05:51:26.854439 26923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:51:26 dist-test-slave-7fr8 krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:26.857945 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:26.847845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:41163 (local address 127.24.255.62:38291)
0926 05:51:26.847925 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:26.847928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:26.848146 (+ 218us) server_negotiation.cc:407] Connection header received
0926 05:51:26.848671 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:26.848673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:26.848688 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:26.848767 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:26.849687 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.850576 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.852064 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.852348 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.852433 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:26.855092 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:26.855094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:26.855094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:26.855101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:26.856470 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:26.856912 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:26.856914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:26.856914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:26.856981 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:26.857292 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:26.857293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:26.857293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:26.857436 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:26.857547 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:26.857805 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 05:51:26.857876 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:51:26.858196 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:26.847761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38291 (local address 127.24.255.2:41163)
0926 05:51:26.848035 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:26.848068 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:26.848547 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:26.848869 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:26.848874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:26.849175 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:26.849555 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:26.849562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.850676 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.850677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:26.851970 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:26.851974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.852470 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.852471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:26.852582 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:26.853186 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:26.853206 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:26.854978 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:26.856568 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:26.856574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:26.856576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:26.856819 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:26.857073 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:26.857074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:26.857075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:26.857185 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:26.857549 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:26.857555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:26.857688 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 05:51:26.857780 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 05:51:26.857914 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":105,"threads_started":1}
I20260926 05:51:26.858706 27054 heartbeater.cc:347] Connected to a master server at 127.24.255.62:38291
I20260926 05:51:26.858795 27054 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:26.858989 27054 heartbeater.cc:511] Master 127.24.255.62:38291 requested a full tablet report, sending...
I20260926 05:51:26.859479 26711 ts_manager.cc:194] Registered new tserver with Master: 66f2951f40f64f098ca4b65e608159f6 (127.24.255.2:43625)
I20260926 05:51:26.859995 26711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:41163
I20260926 05:51:26.863229 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 26923
I20260926 05:51:26.863324 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:51:26.863551 25596 external_mini_cluster.cc:1428] Setting key 7e30c69b2b22be31de00b73ff44e4b27
I20260926 05:51:26.866220 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:26.859836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:45355 (local address 127.24.255.62:38291)
0926 05:51:26.859891 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:26.859893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:26.860001 (+ 108us) server_negotiation.cc:407] Connection header received
0926 05:51:26.860075 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:26.860076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:26.860087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:26.860137 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:26.860455 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.860965 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.861645 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.861850 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.861913 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:26.862975 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:26.862976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:26.862977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:26.862984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:26.864824 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:26.865270 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:26.865271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:26.865271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:26.865326 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:26.865503 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:26.865504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:26.865504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:26.865718 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:26.865795 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:26.865965 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 05:51:26.866083 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:51:26.866556 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:26.859816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38291 (local address 127.24.255.2:45355)
0926 05:51:26.859946 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:26.859957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:26.860018 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:26.860158 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:26.860159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:26.860336 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:26.860388 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:26.860390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.861056 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.861057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:26.861573 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:26.861575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:26.861920 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:26.861920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:26.861996 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:26.862451 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:26.862455 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:26.862861 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:26.864941 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:26.864943 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:26.864943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:26.865189 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:26.865385 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:26.865385 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:26.865386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:26.865450 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:26.865807 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:26.865808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:26.865858 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 05:51:26.865923 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 05:51:26.866056 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:51:26.889863 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38291
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:26.983520 27063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:26.983752 27063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:26.983790 27063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:26.985157 27063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:26.985224 27063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:26.985253 27063 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:51:26.985276 27063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:26.987026 27063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.255.62:38291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:26.987317 27063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:26.987557 27063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:26.993184 27069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:26.993228 27063 server_base.cc:1034] running on GCE node
W20260926 05:51:26.993505 27068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:26.993563 27071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:26.993881 27063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:26.994299 27063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:26.995491 27063 hybrid_clock.cc:648] HybridClock initialized: now 1790401886995423 us; error 95 us; skew 500 ppm
Sep 26 05:51:26 dist-test-slave-7fr8 krb5kdc[26666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:26.998677 27063 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:51:26.999253 27063 webserver.cc:460] Webserver started at http://127.24.255.3:44757/ using document root <none> and password file <none>
I20260926 05:51:26.999495 27063 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:26.999547 27063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:26.999672 27063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:27.000883 27063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "7e61732d35474f2ba8a04da713e3a35a"
format_stamp: "Formatted at 2026-09-26 05:51:26 on dist-test-slave-7fr8"
server_key: "9e0085a4609c641cfae5a60403c53d87"
server_key_iv: "02aadc9d7ef6b519b5402cbb361ec7a8"
server_key_version: "encryptionkey@0"
I20260926 05:51:27.001271 27063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "7e61732d35474f2ba8a04da713e3a35a"
format_stamp: "Formatted at 2026-09-26 05:51:26 on dist-test-slave-7fr8"
server_key: "9e0085a4609c641cfae5a60403c53d87"
server_key_iv: "02aadc9d7ef6b519b5402cbb361ec7a8"
server_key_version: "encryptionkey@0"
I20260926 05:51:27.003077 27063 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 05:51:27.004252 27078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:27.004596 27063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:27.004720 27063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "7e61732d35474f2ba8a04da713e3a35a"
format_stamp: "Formatted at 2026-09-26 05:51:26 on dist-test-slave-7fr8"
server_key: "9e0085a4609c641cfae5a60403c53d87"
server_key_iv: "02aadc9d7ef6b519b5402cbb361ec7a8"
server_key_version: "encryptionkey@0"
I20260926 05:51:27.004802 27063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:27.012372 27063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:27.015165 27063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:27.015324 27063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:27.015936 27063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:27.015998 27063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:27.016031 27063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:27.016047 27063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:51:27 dist-test-slave-7fr8 krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:27.024394 27063 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:44731
I20260926 05:51:27.024458 27193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:44731 every 8 connection(s)
I20260926 05:51:27.024796 27063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:51:27.026165 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 27063
I20260926 05:51:27.026258 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:51:27.026420 25596 external_mini_cluster.cc:1428] Setting key b42aaf8e4ab64e36d0cf8c2e29ef17ad
I20260926 05:51:27.028117 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.016387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:50715 (local address 127.24.255.62:38291)
0926 05:51:27.016447 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:27.016450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:27.016742 (+ 292us) server_negotiation.cc:407] Connection header received
0926 05:51:27.017469 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:27.017470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:27.017519 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:27.017593 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:27.019103 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.019708 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.021307 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.021537 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.021602 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.024657 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:27.024659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:27.024659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:27.024667 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:27.026300 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:27.026765 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:27.026766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:27.026767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:27.026832 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:27.027129 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:27.027130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:27.027130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:27.027462 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:27.027562 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:27.027933 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 05:51:27.028049 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:51:27.028189 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.016431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38291 (local address 127.24.255.3:50715)
0926 05:51:27.016611 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:27.016659 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:27.017313 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:27.017688 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:27.017693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:27.018373 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:27.018976 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.018985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.019820 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.019821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.021172 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.021175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.021623 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.021625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.021813 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.022479 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:27.022488 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:27.024534 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:27.026400 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:27.026407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:27.026409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:27.026704 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:27.026900 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:27.026901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:27.026901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:27.027059 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:27.027583 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:27.027588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:27.027732 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 05:51:27.027841 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 05:51:27.027986 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260926 05:51:27.028767 27194 heartbeater.cc:347] Connected to a master server at 127.24.255.62:38291
I20260926 05:51:27.028857 27194 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:27.029053 27194 heartbeater.cc:511] Master 127.24.255.62:38291 requested a full tablet report, sending...
I20260926 05:51:27.029516 26710 ts_manager.cc:194] Registered new tserver with Master: 7e61732d35474f2ba8a04da713e3a35a (127.24.255.3:44731)
I20260926 05:51:27.030001 26710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:50715
I20260926 05:51:27.038259 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.030005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:48007 (local address 127.24.255.62:38291)
0926 05:51:27.030051 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:27.030053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:27.030174 (+ 121us) server_negotiation.cc:407] Connection header received
0926 05:51:27.030233 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:27.030233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:27.030244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:27.030308 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:27.030663 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.031182 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.032058 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.032252 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.032326 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.034173 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:27.034176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:27.034177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:27.034197 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:27.036334 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:27.036952 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:27.036955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:27.036956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:27.037099 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:27.037470 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:27.037472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:27.037472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:27.037630 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:27.037703 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:27.038040 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 05:51:27.038166 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:51:27.038271 27200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.029937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38291 (local address 127.24.255.3:48007)
0926 05:51:27.030085 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:27.030129 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:27.030183 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:27.030340 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:27.030341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:27.030516 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:27.030580 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.030582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.031373 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.031374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.031945 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.031947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.032452 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.032454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.032687 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.033435 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:27.033442 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:27.033910 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:27.036522 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:27.036524 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:27.036525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:27.036745 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:27.037236 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:27.037237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:27.037237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:27.037317 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:27.037739 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:27.037740 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:27.037819 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 05:51:27.037964 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 05:51:27.038140 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260926 05:51:27.038908 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 05:51:27 dist-test-slave-7fr8 krb5kdc[26666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401887, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:51:27 dist-test-slave-7fr8 krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401887, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:27.067413 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.057671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36970 (local address 127.24.255.62:38291)
0926 05:51:27.057826 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:27.057828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:27.057846 (+ 18us) server_negotiation.cc:407] Connection header received
0926 05:51:27.057877 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:27.057878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:27.057889 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:27.057976 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:27.058840 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.059442 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.060325 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.060529 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.060587 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.063725 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:27.063728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:27.063728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:27.063737 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:27.065468 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:27.065978 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:27.065980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:27.065980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:27.066077 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:27.066589 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:27.066590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:27.066590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:27.066792 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:27.066938 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:27.067185 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 05:51:27.067297 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 05:51:27.069514 26710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:51:27.071303 26710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:27.080088 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.074327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:41231 (local address 127.0.0.1:41738)
0926 05:51:27.074465 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:27.074588 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:27.074649 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:27.074958 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:27.074962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:27.074965 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:27.075052 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.075055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.076337 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.076338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.078750 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.078755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.079741 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.079742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.079806 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.079819 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 05:51:27.079899 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 05:51:27.079990 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":33,"thread_start_us":133,"threads_started":1}
I20260926 05:51:27.080201 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.074327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:44731 (local address 127.0.0.1:54390)
0926 05:51:27.074538 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:27.074552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:27.074618 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:27.075379 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:27.075380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:27.075383 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:27.075423 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.075424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.076703 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.076704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.078077 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.078082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.079661 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.079662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.079771 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.080043 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 05:51:27.080072 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 05:51:27.080125 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":27,"thread_start_us":71,"threads_started":1}
I20260926 05:51:27.080152 27216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.074760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.24.255.2:43625)
0926 05:51:27.075108 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:27.075112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:27.075121 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:27.075168 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:27.075170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:27.075186 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:27.075254 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:27.075445 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.076592 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.079169 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.079808 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.079884 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.079954 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:27.080018 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 05:51:27.080061 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":112,"threads_started":1}
I20260926 05:51:27.080461 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.074611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.24.255.1:41231)
0926 05:51:27.074772 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:27.074776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:27.074783 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:27.074814 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:27.074816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:27.074828 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:27.074911 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:27.075124 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.076105 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.078838 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.079646 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.079719 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.079789 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:27.080238 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 05:51:27.080374 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260926 05:51:27.080569 27213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.074425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.24.255.3:44731)
0926 05:51:27.075209 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:27.075213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:27.075221 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:27.075274 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:27.075276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:27.075291 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:27.075363 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:27.075484 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.076577 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.078404 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.079504 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.080325 (+ 821us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.080404 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:27.080432 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 05:51:27.080479 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":82,"threads_started":1}
I20260926 05:51:27.081132 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.074552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:43625 (local address 127.0.0.1:48450)
0926 05:51:27.074958 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:27.074968 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:27.075017 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:27.075316 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:27.075317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:27.075319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:27.075374 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.075376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.077643 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.077644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.079071 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.079074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.079885 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.079886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.079952 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.079957 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:27.080021 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:51:27.080875 (+ 854us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":62,"threads_started":1}
I20260926 05:51:27.081163 26848 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae974112aff4bce8faae09b5076f505 (DEFAULT_TABLE table=test-table [id=a147105611d34b44ba14253d8d2b894f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:27.081530 26848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae974112aff4bce8faae09b5076f505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:27.081503 27128 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae974112aff4bce8faae09b5076f505 (DEFAULT_TABLE table=test-table [id=a147105611d34b44ba14253d8d2b894f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:27.081804 27128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae974112aff4bce8faae09b5076f505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:27.081948 26988 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae974112aff4bce8faae09b5076f505 (DEFAULT_TABLE table=test-table [id=a147105611d34b44ba14253d8d2b894f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:27.082283 26988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae974112aff4bce8faae09b5076f505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:27.084697 27217 tablet_bootstrap.cc:492] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a: Bootstrap starting.
I20260926 05:51:27.085561 27217 tablet_bootstrap.cc:654] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:27.085642 27219 tablet_bootstrap.cc:492] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6: Bootstrap starting.
I20260926 05:51:27.085856 27217 log.cc:824] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:27.085865 27218 tablet_bootstrap.cc:492] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927: Bootstrap starting.
I20260926 05:51:27.086316 27219 tablet_bootstrap.cc:654] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:27.086484 27218 tablet_bootstrap.cc:654] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:27.086651 27219 log.cc:824] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:27.086803 27218 log.cc:824] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:27.087314 27217 tablet_bootstrap.cc:492] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a: No bootstrap required, opened a new log
I20260926 05:51:27.087435 27217 ts_tablet_manager.cc:1397] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:51:27.087556 27219 tablet_bootstrap.cc:492] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6: No bootstrap required, opened a new log
I20260926 05:51:27.087643 27219 ts_tablet_manager.cc:1397] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:51:27.087687 27218 tablet_bootstrap.cc:492] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927: No bootstrap required, opened a new log
I20260926 05:51:27.087740 27218 ts_tablet_manager.cc:1397] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:51:27.089936 27219 raft_consensus.cc:348] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } }
I20260926 05:51:27.089965 27217 raft_consensus.cc:348] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } }
I20260926 05:51:27.090168 27217 raft_consensus.cc:374] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:27.090170 27219 raft_consensus.cc:374] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:27.090214 27217 raft_consensus.cc:729] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e61732d35474f2ba8a04da713e3a35a, State: Initialized, Role: FOLLOWER
I20260926 05:51:27.090214 27219 raft_consensus.cc:729] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f2951f40f64f098ca4b65e608159f6, State: Initialized, Role: FOLLOWER
I20260926 05:51:27.090179 27218 raft_consensus.cc:348] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } }
I20260926 05:51:27.090346 27218 raft_consensus.cc:374] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:27.090381 27218 raft_consensus.cc:729] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067e6de0cd3d4506b77e97217e56c927, State: Initialized, Role: FOLLOWER
I20260926 05:51:27.090346 27217 consensus_queue.cc:260] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } }
I20260926 05:51:27.090389 27219 consensus_queue.cc:260] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } }
I20260926 05:51:27.090481 27218 consensus_queue.cc:260] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } }
I20260926 05:51:27.090672 27217 ts_tablet_manager.cc:1428] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 05:51:27.090726 27219 ts_tablet_manager.cc:1428] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:51:27.090808 27194 heartbeater.cc:503] Master 127.24.255.62:38291 was elected leader, sending a full tablet report...
I20260926 05:51:27.090837 27054 heartbeater.cc:503] Master 127.24.255.62:38291 was elected leader, sending a full tablet report...
I20260926 05:51:27.090726 27218 ts_tablet_manager.cc:1428] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 05:51:27.091372 26914 heartbeater.cc:503] Master 127.24.255.62:38291 was elected leader, sending a full tablet report...
W20260926 05:51:27.105162 27055 tablet.cc:2369] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:51:27.191726 26915 tablet.cc:2369] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:51:27.275743 27195 tablet.cc:2369] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:51:27.300482 27225 raft_consensus.cc:482] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:27.300828 27225 raft_consensus.cc:504] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } }
I20260926 05:51:27.301235 27225 leader_election.cc:288] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e61732d35474f2ba8a04da713e3a35a (127.24.255.3:44731), 66f2951f40f64f098ca4b65e608159f6 (127.24.255.2:43625)
I20260926 05:51:27.306016 27216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.301613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:40021 (local address 127.24.255.2:43625)
0926 05:51:27.301659 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:27.301661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:27.301673 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:51:27.301740 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:27.301753 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:27.301763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:27.301833 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:27.302147 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.303029 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.304764 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.305352 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.305456 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.305468 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:27.305859 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 05:51:27.305916 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:51:27.306149 27227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.301475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:44731 (local address 127.24.255.1:53421)
0926 05:51:27.301607 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:27.301619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:27.301673 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:27.301878 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:27.301880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:27.301883 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:27.302038 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.302039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.303206 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.303207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.304610 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.304613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.305563 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.305565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.305726 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.305739 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 05:51:27.305835 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 05:51:27.305959 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":103,"thread_start_us":63,"threads_started":1}
I20260926 05:51:27.306216 27213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.301628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:53421 (local address 127.24.255.3:44731)
0926 05:51:27.301671 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:27.301673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:27.301681 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:27.301779 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:27.301780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:27.301787 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:27.301836 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:27.302147 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.303028 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.304763 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.305390 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.305474 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.305482 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:27.306039 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 05:51:27.306145 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:51:27.306465 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.301477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:43625 (local address 127.24.255.1:40021)
0926 05:51:27.301607 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:27.301619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:27.301680 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:27.301878 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:27.301879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:27.301884 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:27.302005 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.302007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.303209 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.303210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.304610 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.304613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.305562 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.305565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.305726 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.305738 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 05:51:27.305835 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 05:51:27.305959 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":168,"thread_start_us":59,"threads_started":1}
I20260926 05:51:27.306704 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae974112aff4bce8faae09b5076f505" candidate_uuid: "067e6de0cd3d4506b77e97217e56c927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61732d35474f2ba8a04da713e3a35a" is_pre_election: true
I20260926 05:51:27.306932 27134 raft_consensus.cc:2454] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067e6de0cd3d4506b77e97217e56c927 in term 0.
I20260926 05:51:27.306986 27008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae974112aff4bce8faae09b5076f505" candidate_uuid: "067e6de0cd3d4506b77e97217e56c927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f2951f40f64f098ca4b65e608159f6" is_pre_election: true
I20260926 05:51:27.307150 27008 raft_consensus.cc:2454] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067e6de0cd3d4506b77e97217e56c927 in term 0.
I20260926 05:51:27.307298 26799 leader_election.cc:302] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067e6de0cd3d4506b77e97217e56c927, 7e61732d35474f2ba8a04da713e3a35a; no voters:
I20260926 05:51:27.307516 27225 raft_consensus.cc:2781] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:27.307574 27225 raft_consensus.cc:482] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:27.307588 27225 raft_consensus.cc:3037] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:27.308552 27225 raft_consensus.cc:504] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } }
I20260926 05:51:27.308729 27225 leader_election.cc:288] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [CANDIDATE]: Term 1 election: Requested vote from peers 7e61732d35474f2ba8a04da713e3a35a (127.24.255.3:44731), 66f2951f40f64f098ca4b65e608159f6 (127.24.255.2:43625)
I20260926 05:51:27.308934 27008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae974112aff4bce8faae09b5076f505" candidate_uuid: "067e6de0cd3d4506b77e97217e56c927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f2951f40f64f098ca4b65e608159f6"
I20260926 05:51:27.309077 27008 raft_consensus.cc:3037] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:27.309115 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae974112aff4bce8faae09b5076f505" candidate_uuid: "067e6de0cd3d4506b77e97217e56c927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61732d35474f2ba8a04da713e3a35a"
I20260926 05:51:27.309242 27134 raft_consensus.cc:3037] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:27.309960 27008 raft_consensus.cc:2454] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067e6de0cd3d4506b77e97217e56c927 in term 1.
I20260926 05:51:27.310002 27134 raft_consensus.cc:2454] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067e6de0cd3d4506b77e97217e56c927 in term 1.
I20260926 05:51:27.310178 26800 leader_election.cc:302] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067e6de0cd3d4506b77e97217e56c927, 66f2951f40f64f098ca4b65e608159f6; no voters:
I20260926 05:51:27.310294 27225 raft_consensus.cc:2781] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:27.310451 27225 raft_consensus.cc:686] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [term 1 LEADER]: Becoming Leader. State: Replica: 067e6de0cd3d4506b77e97217e56c927, State: Running, Role: LEADER
I20260926 05:51:27.310550 27225 consensus_queue.cc:237] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } }
I20260926 05:51:27.311468 26709 catalog_manager.cc:5495] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 reported cstate change: term changed from 0 to 1, leader changed from <none> to 067e6de0cd3d4506b77e97217e56c927 (127.24.255.1). New cstate: current_term: 1 leader_uuid: "067e6de0cd3d4506b77e97217e56c927" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } health_report { overall_health: UNKNOWN } } }
I20260926 05:51:27.369050 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.366063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.24.255.1:41231)
0926 05:51:27.366142 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:27.366144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:27.366154 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:27.366189 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:27.366190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:27.366208 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:27.366274 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:27.366502 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.367071 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.368050 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.368271 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.368336 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.368671 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:27.368775 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:27.368831 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:27.368918 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 05:51:27.368939 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 05:51:27.370980 27134 raft_consensus.cc:1261] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a [term 1 FOLLOWER]: Refusing update from remote peer 067e6de0cd3d4506b77e97217e56c927: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:27.370980 27008 raft_consensus.cc:1261] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6 [term 1 FOLLOWER]: Refusing update from remote peer 067e6de0cd3d4506b77e97217e56c927: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:27.371301 27225 consensus_queue.cc:1035] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:27.371495 27228 consensus_queue.cc:1035] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:27.373314 27231 mvcc.cc:205] Tried to move back new op lower bound from 7333486130668654592 to 7333486130424684544. Current Snapshot: MvccSnapshot[applied={T|T < 7333486130668654592}]
I20260926 05:51:27.374259 27232 mvcc.cc:205] Tried to move back new op lower bound from 7333486130668654592 to 7333486130424684544. Current Snapshot: MvccSnapshot[applied={T|T < 7333486130668654592}]
I20260926 05:51:27.374637 27234 mvcc.cc:205] Tried to move back new op lower bound from 7333486130668654592 to 7333486130424684544. Current Snapshot: MvccSnapshot[applied={T|T < 7333486130668654592}]
I20260926 05:51:27.377532 26711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:27.377708 26711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:27.379482 26711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327: Sending DeleteTablet for 3 replicas of tablet 8ae974112aff4bce8faae09b5076f505
I20260926 05:51:27.379992 27128 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae974112aff4bce8faae09b5076f505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:27 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:54390
I20260926 05:51:27.379936 26848 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae974112aff4bce8faae09b5076f505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:27 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:41738
I20260926 05:51:27.380215 26988 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae974112aff4bce8faae09b5076f505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:27 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:48450
I20260926 05:51:27.380367 26711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:51:27.380529 26711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:27.380730 27240 tablet_replica.cc:330] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a: stopping tablet replica
I20260926 05:51:27.380865 27242 tablet_replica.cc:330] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6: stopping tablet replica
I20260926 05:51:27.380870 27240 raft_consensus.cc:2229] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:27.381012 27240 raft_consensus.cc:2258] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:27.381139 27242 raft_consensus.cc:2229] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:27.381188 27241 tablet_replica.cc:330] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927: stopping tablet replica
I20260926 05:51:27.381278 27242 raft_consensus.cc:2258] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:27.381318 27241 raft_consensus.cc:2229] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:51:27.381446 27241 raft_consensus.cc:2258] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:27.381543 27240 ts_tablet_manager.cc:1905] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:27.381727 27242 ts_tablet_manager.cc:1905] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:27.382843 26848 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd4b70e3bfa4286aba4417638f18cb3 (DEFAULT_TABLE table=test-table [id=a011f1e1b1924741b830f7263ede6cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:27.382954 26848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd4b70e3bfa4286aba4417638f18cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:27.383548 26988 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd4b70e3bfa4286aba4417638f18cb3 (DEFAULT_TABLE table=test-table [id=a011f1e1b1924741b830f7263ede6cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:27.383853 27240 ts_tablet_manager.cc:1918] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:27.383992 27240 log.cc:1196] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/8ae974112aff4bce8faae09b5076f505
I20260926 05:51:27.383989 27241 ts_tablet_manager.cc:1905] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:27.384347 27240 ts_tablet_manager.cc:1939] T 8ae974112aff4bce8faae09b5076f505 P 7e61732d35474f2ba8a04da713e3a35a: Deleting consensus metadata
I20260926 05:51:27.384407 27218 tablet_bootstrap.cc:492] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927: Bootstrap starting.
I20260926 05:51:27.384804 27218 tablet_bootstrap.cc:654] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:27.384828 26988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd4b70e3bfa4286aba4417638f18cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:27.385330 26695 catalog_manager.cc:4841] TS 7e61732d35474f2ba8a04da713e3a35a (127.24.255.3:44731): tablet 8ae974112aff4bce8faae09b5076f505 (table test-table [id=a147105611d34b44ba14253d8d2b894f]) successfully deleted
I20260926 05:51:27.385550 27241 ts_tablet_manager.cc:1918] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:27.385614 27241 log.cc:1196] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/wal/wals/8ae974112aff4bce8faae09b5076f505
I20260926 05:51:27.385800 27128 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd4b70e3bfa4286aba4417638f18cb3 (DEFAULT_TABLE table=test-table [id=a011f1e1b1924741b830f7263ede6cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:27.385859 27241 ts_tablet_manager.cc:1939] T 8ae974112aff4bce8faae09b5076f505 P 067e6de0cd3d4506b77e97217e56c927: Deleting consensus metadata
I20260926 05:51:27.385905 27128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd4b70e3bfa4286aba4417638f18cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:27.386055 27242 ts_tablet_manager.cc:1918] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:27.386274 27242 log.cc:1196] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/8ae974112aff4bce8faae09b5076f505
I20260926 05:51:27.386521 27242 ts_tablet_manager.cc:1939] T 8ae974112aff4bce8faae09b5076f505 P 66f2951f40f64f098ca4b65e608159f6: Deleting consensus metadata
I20260926 05:51:27.386972 27219 tablet_bootstrap.cc:492] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6: Bootstrap starting.
I20260926 05:51:27.386965 26696 catalog_manager.cc:4841] TS 067e6de0cd3d4506b77e97217e56c927 (127.24.255.1:41231): tablet 8ae974112aff4bce8faae09b5076f505 (table test-table [id=a147105611d34b44ba14253d8d2b894f]) successfully deleted
I20260926 05:51:27.387039 26696 catalog_manager.cc:4841] TS 66f2951f40f64f098ca4b65e608159f6 (127.24.255.2:43625): tablet 8ae974112aff4bce8faae09b5076f505 (table test-table [id=a147105611d34b44ba14253d8d2b894f]) successfully deleted
I20260926 05:51:27.387053 27217 tablet_bootstrap.cc:492] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a: Bootstrap starting.
I20260926 05:51:27.387336 27219 tablet_bootstrap.cc:654] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:27.387356 27217 tablet_bootstrap.cc:654] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:27.387914 27217 tablet_bootstrap.cc:492] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a: No bootstrap required, opened a new log
I20260926 05:51:27.387977 27217 ts_tablet_manager.cc:1397] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:27.388129 27217 raft_consensus.cc:348] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.388191 27217 raft_consensus.cc:374] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:27.388211 27217 raft_consensus.cc:729] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e61732d35474f2ba8a04da713e3a35a, State: Initialized, Role: FOLLOWER
I20260926 05:51:27.388266 27217 consensus_queue.cc:260] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.388367 27217 ts_tablet_manager.cc:1428] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:27.388412 27218 tablet_bootstrap.cc:492] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927: No bootstrap required, opened a new log
I20260926 05:51:27.388453 27218 ts_tablet_manager.cc:1397] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s
I20260926 05:51:27.388583 27218 raft_consensus.cc:348] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.388661 27218 raft_consensus.cc:374] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:27.388682 27218 raft_consensus.cc:729] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067e6de0cd3d4506b77e97217e56c927, State: Initialized, Role: FOLLOWER
I20260926 05:51:27.388717 27218 consensus_queue.cc:260] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.388818 27218 ts_tablet_manager.cc:1428] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:27.388886 27219 tablet_bootstrap.cc:492] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6: No bootstrap required, opened a new log
I20260926 05:51:27.388936 27219 ts_tablet_manager.cc:1397] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:51:27.389307 27219 raft_consensus.cc:348] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.389457 27219 raft_consensus.cc:374] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:27.389487 27219 raft_consensus.cc:729] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f2951f40f64f098ca4b65e608159f6, State: Initialized, Role: FOLLOWER
I20260926 05:51:27.389528 27219 consensus_queue.cc:260] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.389612 27219 ts_tablet_manager.cc:1428] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:27.867457 27224 raft_consensus.cc:482] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:27.867604 27224 raft_consensus.cc:504] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.867928 27224 leader_election.cc:288] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 067e6de0cd3d4506b77e97217e56c927 (127.24.255.1:41231), 7e61732d35474f2ba8a04da713e3a35a (127.24.255.3:44731)
I20260926 05:51:27.870899 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.868203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:50815 (local address 127.24.255.1:41231)
0926 05:51:27.868262 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:27.868265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:27.868315 (+ 50us) server_negotiation.cc:407] Connection header received
0926 05:51:27.868397 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:27.868398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:27.868410 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:27.868461 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:27.868587 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.869166 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.870182 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.870556 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.870620 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.870629 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:27.870777 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 05:51:27.870824 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:51:27.870926 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.868279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:44155 (local address 127.24.255.3:44731)
0926 05:51:27.868392 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:27.868393 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:27.868402 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:27.868427 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:27.868428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:27.868437 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:27.868496 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:27.868626 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.869179 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.870177 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.870569 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.870631 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.870640 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:27.870794 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 05:51:27.870844 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
I20260926 05:51:27.870950 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.868137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:41231 (local address 127.24.255.2:50815)
0926 05:51:27.868258 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:27.868268 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:27.868330 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:27.868469 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:27.868470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:27.868472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:27.868529 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.868531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.869250 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.869251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.870078 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.870081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.870651 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.870652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.870716 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.870720 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:51:27.870766 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:51:27.870824 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
I20260926 05:51:27.870955 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.868190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:44731 (local address 127.24.255.2:44155)
0926 05:51:27.868300 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:27.868310 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:27.868365 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:27.868508 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:27.868509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:27.868511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:27.868561 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.868563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.869278 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.869279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.870083 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.870085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.870676 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.870677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.870728 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.870731 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:51:27.870776 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 05:51:27.870827 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":24,"thread_start_us":53,"threads_started":1}
I20260926 05:51:27.871253 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd4b70e3bfa4286aba4417638f18cb3" candidate_uuid: "66f2951f40f64f098ca4b65e608159f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61732d35474f2ba8a04da713e3a35a" is_pre_election: true
I20260926 05:51:27.871271 26868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd4b70e3bfa4286aba4417638f18cb3" candidate_uuid: "66f2951f40f64f098ca4b65e608159f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067e6de0cd3d4506b77e97217e56c927" is_pre_election: true
I20260926 05:51:27.871356 27134 raft_consensus.cc:2454] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f2951f40f64f098ca4b65e608159f6 in term 0.
I20260926 05:51:27.871412 26868 raft_consensus.cc:2454] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f2951f40f64f098ca4b65e608159f6 in term 0.
I20260926 05:51:27.871555 26939 leader_election.cc:302] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f2951f40f64f098ca4b65e608159f6, 7e61732d35474f2ba8a04da713e3a35a; no voters:
I20260926 05:51:27.871696 27224 raft_consensus.cc:2781] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:27.871743 27224 raft_consensus.cc:482] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:27.871757 27224 raft_consensus.cc:3037] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:27.872470 27224 raft_consensus.cc:504] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.872668 27224 leader_election.cc:288] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [CANDIDATE]: Term 1 election: Requested vote from peers 067e6de0cd3d4506b77e97217e56c927 (127.24.255.1:41231), 7e61732d35474f2ba8a04da713e3a35a (127.24.255.3:44731)
I20260926 05:51:27.872820 26868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd4b70e3bfa4286aba4417638f18cb3" candidate_uuid: "66f2951f40f64f098ca4b65e608159f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067e6de0cd3d4506b77e97217e56c927"
I20260926 05:51:27.872833 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd4b70e3bfa4286aba4417638f18cb3" candidate_uuid: "66f2951f40f64f098ca4b65e608159f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61732d35474f2ba8a04da713e3a35a"
I20260926 05:51:27.872891 26868 raft_consensus.cc:3037] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:27.872901 27134 raft_consensus.cc:3037] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:27.873492 27134 raft_consensus.cc:2454] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f2951f40f64f098ca4b65e608159f6 in term 1.
I20260926 05:51:27.873572 26868 raft_consensus.cc:2454] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f2951f40f64f098ca4b65e608159f6 in term 1.
I20260926 05:51:27.873672 26939 leader_election.cc:302] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f2951f40f64f098ca4b65e608159f6, 7e61732d35474f2ba8a04da713e3a35a; no voters:
I20260926 05:51:27.873843 27224 raft_consensus.cc:2781] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:27.874027 27224 raft_consensus.cc:686] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [term 1 LEADER]: Becoming Leader. State: Replica: 66f2951f40f64f098ca4b65e608159f6, State: Running, Role: LEADER
I20260926 05:51:27.874132 27224 consensus_queue.cc:237] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.874822 26711 catalog_manager.cc:5495] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66f2951f40f64f098ca4b65e608159f6 (127.24.255.2). New cstate: current_term: 1 leader_uuid: "66f2951f40f64f098ca4b65e608159f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } health_report { overall_health: UNKNOWN } } }
Sep 26 05:51:27 dist-test-slave-7fr8 krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:27.966612 27256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.959097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38291 (local address 127.0.0.1:36978)
0926 05:51:27.959331 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:27.959342 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:27.959439 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:27.959592 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:27.959593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:27.959908 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:27.960072 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.960075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.960759 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.960760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.961277 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:27.961278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.961606 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.961607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:27.961662 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.962387 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:27.962392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:27.963794 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:27.965562 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:27.965571 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:27.965574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:27.965766 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:27.965949 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:27.965950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:27.965950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:27.966000 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:27.966266 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:27.966273 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:27.966317 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 05:51:27.966378 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:51:27.966494 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":119,"threads_started":1}
I20260926 05:51:27.966663 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:27.959136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36978 (local address 127.24.255.62:38291)
0926 05:51:27.959324 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:27.959326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:27.959416 (+ 90us) server_negotiation.cc:407] Connection header received
0926 05:51:27.959489 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:27.959490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:27.959501 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:27.959589 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:27.960158 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.960683 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.961354 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:27.961547 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:27.961600 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:27.963894 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:27.963896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:27.963896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:27.963904 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:27.965468 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:27.965831 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:27.965832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:27.965832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:27.965895 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:27.966047 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:27.966048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:27.966048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:27.966213 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:27.966271 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:27.966406 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 05:51:27.966555 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":156,"thread_start_us":126,"threads_started":1}
I20260926 05:51:27.968883 26711 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:36978:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 05:51:27.969125 26711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:27.970685 26988 tablet_service.cc:1467] Processing CreateTablet for tablet 795b77eefc764febb8942273f7bd9b43 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f6ba5914f3c141c1adac47fbc8d9655a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:51:27.970677 26848 tablet_service.cc:1467] Processing CreateTablet for tablet 795b77eefc764febb8942273f7bd9b43 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f6ba5914f3c141c1adac47fbc8d9655a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:51:27.970841 26848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795b77eefc764febb8942273f7bd9b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:27.970840 26988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795b77eefc764febb8942273f7bd9b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:27.972610 27258 tablet_bootstrap.cc:492] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927: Bootstrap starting.
I20260926 05:51:27.972674 27128 tablet_service.cc:1467] Processing CreateTablet for tablet 795b77eefc764febb8942273f7bd9b43 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f6ba5914f3c141c1adac47fbc8d9655a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:51:27.972823 27128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795b77eefc764febb8942273f7bd9b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:27.973083 27258 tablet_bootstrap.cc:654] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:27.973385 27259 tablet_bootstrap.cc:492] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6: Bootstrap starting.
I20260926 05:51:27.973789 27259 tablet_bootstrap.cc:654] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:27.973930 27258 tablet_bootstrap.cc:492] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927: No bootstrap required, opened a new log
I20260926 05:51:27.974025 27258 ts_tablet_manager.cc:1397] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:27.974153 27261 tablet_bootstrap.cc:492] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a: Bootstrap starting.
I20260926 05:51:27.974241 27258 raft_consensus.cc:348] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.974321 27258 raft_consensus.cc:374] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:27.974350 27258 raft_consensus.cc:729] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067e6de0cd3d4506b77e97217e56c927, State: Initialized, Role: FOLLOWER
I20260926 05:51:27.974412 27258 consensus_queue.cc:260] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.974529 27261 tablet_bootstrap.cc:654] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:27.974558 27258 ts_tablet_manager.cc:1428] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:27.974637 27259 tablet_bootstrap.cc:492] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6: No bootstrap required, opened a new log
I20260926 05:51:27.974677 27259 ts_tablet_manager.cc:1397] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:27.974785 27259 raft_consensus.cc:348] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.974866 27259 raft_consensus.cc:374] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:27.974890 27259 raft_consensus.cc:729] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f2951f40f64f098ca4b65e608159f6, State: Initialized, Role: FOLLOWER
I20260926 05:51:27.974952 27259 consensus_queue.cc:260] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.975097 27261 tablet_bootstrap.cc:492] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a: No bootstrap required, opened a new log
I20260926 05:51:27.975104 27224 tablet_replica.cc:439] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } }
I20260926 05:51:27.975147 27261 ts_tablet_manager.cc:1397] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:27.975162 27224 tablet_replica.cc:442] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:27.975241 27259 ts_tablet_manager.cc:1428] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 05:51:27.975281 27261 raft_consensus.cc:348] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.975363 27261 raft_consensus.cc:374] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:27.975383 27261 raft_consensus.cc:729] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e61732d35474f2ba8a04da713e3a35a, State: Initialized, Role: FOLLOWER
I20260926 05:51:27.975359 27263 tablet_replica.cc:439] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } }
I20260926 05:51:27.975425 27261 consensus_queue.cc:260] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:27.975468 27263 tablet_replica.cc:442] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:27.975548 27261 ts_tablet_manager.cc:1428] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:51:27.975608 27265 tablet_replica.cc:439] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } }
I20260926 05:51:27.975665 27265 tablet_replica.cc:442] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:28.288259 27224 raft_consensus.cc:482] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:28.288360 27224 raft_consensus.cc:504] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:28.288527 27224 leader_election.cc:288] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 067e6de0cd3d4506b77e97217e56c927 (127.24.255.1:41231), 7e61732d35474f2ba8a04da713e3a35a (127.24.255.3:44731)
I20260926 05:51:28.288777 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795b77eefc764febb8942273f7bd9b43" candidate_uuid: "66f2951f40f64f098ca4b65e608159f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61732d35474f2ba8a04da713e3a35a" is_pre_election: true
I20260926 05:51:28.288762 26868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795b77eefc764febb8942273f7bd9b43" candidate_uuid: "66f2951f40f64f098ca4b65e608159f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067e6de0cd3d4506b77e97217e56c927" is_pre_election: true
I20260926 05:51:28.288880 26868 raft_consensus.cc:2454] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f2951f40f64f098ca4b65e608159f6 in term 0.
I20260926 05:51:28.288870 27134 raft_consensus.cc:2454] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f2951f40f64f098ca4b65e608159f6 in term 0.
I20260926 05:51:28.289105 26940 leader_election.cc:302] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 067e6de0cd3d4506b77e97217e56c927, 66f2951f40f64f098ca4b65e608159f6, 7e61732d35474f2ba8a04da713e3a35a; no voters:
I20260926 05:51:28.289223 27224 raft_consensus.cc:2781] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:28.289273 27224 raft_consensus.cc:482] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:28.289292 27224 raft_consensus.cc:3037] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:28.289929 27224 raft_consensus.cc:504] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:28.290056 27224 leader_election.cc:288] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [CANDIDATE]: Term 1 election: Requested vote from peers 067e6de0cd3d4506b77e97217e56c927 (127.24.255.1:41231), 7e61732d35474f2ba8a04da713e3a35a (127.24.255.3:44731)
I20260926 05:51:28.290266 27134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795b77eefc764febb8942273f7bd9b43" candidate_uuid: "66f2951f40f64f098ca4b65e608159f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e61732d35474f2ba8a04da713e3a35a"
I20260926 05:51:28.290343 27134 raft_consensus.cc:3037] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:28.290478 26868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795b77eefc764febb8942273f7bd9b43" candidate_uuid: "66f2951f40f64f098ca4b65e608159f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067e6de0cd3d4506b77e97217e56c927"
I20260926 05:51:28.290611 26868 raft_consensus.cc:3037] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:28.291055 27134 raft_consensus.cc:2454] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f2951f40f64f098ca4b65e608159f6 in term 1.
I20260926 05:51:28.291256 26939 leader_election.cc:302] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f2951f40f64f098ca4b65e608159f6, 7e61732d35474f2ba8a04da713e3a35a; no voters:
I20260926 05:51:28.291357 27224 raft_consensus.cc:2781] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:28.291419 27224 raft_consensus.cc:686] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [term 1 LEADER]: Becoming Leader. State: Replica: 66f2951f40f64f098ca4b65e608159f6, State: Running, Role: LEADER
I20260926 05:51:28.291474 27224 consensus_queue.cc:237] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } }
I20260926 05:51:28.291765 27249 tablet_replica.cc:439] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6: TxnStatusTablet state changed. Reason: New leader 66f2951f40f64f098ca4b65e608159f6. Latest consensus state: current_term: 1 leader_uuid: "66f2951f40f64f098ca4b65e608159f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } }
I20260926 05:51:28.291843 27249 tablet_replica.cc:442] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:51:28.291828 26868 raft_consensus.cc:2454] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f2951f40f64f098ca4b65e608159f6 in term 1.
I20260926 05:51:28.291998 27267 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 05:51:28.292145 27267 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 05:51:28.292201 26709 catalog_manager.cc:5495] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66f2951f40f64f098ca4b65e608159f6 (127.24.255.2). New cstate: current_term: 1 leader_uuid: "66f2951f40f64f098ca4b65e608159f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } health_report { overall_health: UNKNOWN } } }
I20260926 05:51:28.334280 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:28.331143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48460 (local address 127.24.255.2:43625)
0926 05:51:28.331340 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:28.331341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:28.331350 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:28.331388 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:28.331388 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:28.331408 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:28.331483 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:28.331631 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.332210 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.333344 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.333571 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.333636 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:28.333887 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:28.333999 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:28.334066 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:28.334174 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 05:51:28.334198 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260926 05:51:28.334316 27256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:28.330883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:43625 (local address 127.0.0.1:48460)
0926 05:51:28.330971 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:28.330985 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:28.331043 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:28.331507 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:28.331508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:28.331510 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:51:28.331567 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:28.331569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.332284 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.332285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:28.333220 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:28.333223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.333645 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.333646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:28.333714 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:28.333736 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:51:28.334064 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:51:28.334069 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:28.334161 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 05:51:28.334219 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 05:51:28.335544 27134 raft_consensus.cc:1261] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a [term 1 FOLLOWER]: Refusing update from remote peer 66f2951f40f64f098ca4b65e608159f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:28.335594 26868 raft_consensus.cc:1261] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927 [term 1 FOLLOWER]: Refusing update from remote peer 66f2951f40f64f098ca4b65e608159f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:28.335824 27249 consensus_queue.cc:1035] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:28.335999 27224 consensus_queue.cc:1035] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:28.336797 27263 tablet_replica.cc:439] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927: TxnStatusTablet state changed. Reason: New leader 66f2951f40f64f098ca4b65e608159f6. Latest consensus state: current_term: 1 leader_uuid: "66f2951f40f64f098ca4b65e608159f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } }
I20260926 05:51:28.336902 27263 tablet_replica.cc:442] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:28.337102 27265 tablet_replica.cc:439] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a: TxnStatusTablet state changed. Reason: New leader 66f2951f40f64f098ca4b65e608159f6. Latest consensus state: current_term: 1 leader_uuid: "66f2951f40f64f098ca4b65e608159f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } }
I20260926 05:51:28.337208 27265 tablet_replica.cc:442] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:28.337952 27224 tablet_replica.cc:439] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66f2951f40f64f098ca4b65e608159f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } }
I20260926 05:51:28.337965 27249 tablet_replica.cc:439] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66f2951f40f64f098ca4b65e608159f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } }
I20260926 05:51:28.338052 27224 tablet_replica.cc:442] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:51:28.338064 27249 tablet_replica.cc:442] T 795b77eefc764febb8942273f7bd9b43 P 66f2951f40f64f098ca4b65e608159f6: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:51:28.338313 27265 tablet_replica.cc:439] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66f2951f40f64f098ca4b65e608159f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } }
I20260926 05:51:28.338397 27265 tablet_replica.cc:442] T 795b77eefc764febb8942273f7bd9b43 P 7e61732d35474f2ba8a04da713e3a35a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:28.338515 27263 tablet_replica.cc:439] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66f2951f40f64f098ca4b65e608159f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 } } peers { permanent_uuid: "66f2951f40f64f098ca4b65e608159f6" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43625 } } peers { permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 } } }
I20260926 05:51:28.338604 27263 tablet_replica.cc:442] T 795b77eefc764febb8942273f7bd9b43 P 067e6de0cd3d4506b77e97217e56c927: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:28.342656 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:28.340369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48468 (local address 127.24.255.2:43625)
0926 05:51:28.340407 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:28.340409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:28.340440 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:51:28.340505 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:28.340505 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:28.340516 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:28.340558 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:28.340721 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.341212 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.341955 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.342217 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.342268 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:28.342419 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:28.342454 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:28.342498 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:28.342557 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 05:51:28.342578 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:51:28.350138 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:28.346917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:55233 (local address 127.24.255.2:43625)
0926 05:51:28.346957 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:28.346983 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:28.347101 (+ 118us) server_negotiation.cc:407] Connection header received
0926 05:51:28.347195 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:28.347196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:28.347204 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:28.347229 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:28.347422 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.348189 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.349296 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.349490 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.349596 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:28.349816 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:28.349850 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:28.349904 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:28.349972 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 05:51:28.349994 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:51:28.350164 27284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:28.346819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:43625 (local address 127.24.255.2:55233)
0926 05:51:28.346982 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:28.347017 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:28.347107 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:28.347284 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:28.347284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:28.347286 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:51:28.347353 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:28.347355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.348273 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.348274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:28.349209 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:28.349212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.349606 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.349607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:28.349679 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:28.349699 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:51:28.349916 (+ 217us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:51:28.349918 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:51:28.349966 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:51:28.350012 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260926 05:51:28.352708 27134 raft_consensus.cc:1261] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a [term 1 FOLLOWER]: Refusing update from remote peer 66f2951f40f64f098ca4b65e608159f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:28.352718 26868 raft_consensus.cc:1261] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927 [term 1 FOLLOWER]: Refusing update from remote peer 66f2951f40f64f098ca4b65e608159f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:28.352927 27276 consensus_queue.cc:1035] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e61732d35474f2ba8a04da713e3a35a" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:28.353068 27224 consensus_queue.cc:1035] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067e6de0cd3d4506b77e97217e56c927" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:28.369654 26711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:28.369786 26711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:28.372181 26711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 547623e2a679479fbf52c9272afaa327: Sending DeleteTablet for 3 replicas of tablet 8bd4b70e3bfa4286aba4417638f18cb3
I20260926 05:51:28.372586 26987 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd4b70e3bfa4286aba4417638f18cb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:28 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:48450
I20260926 05:51:28.372576 27128 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd4b70e3bfa4286aba4417638f18cb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:28 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:54390
I20260926 05:51:28.372776 27288 tablet_replica.cc:330] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6: stopping tablet replica
I20260926 05:51:28.372840 27288 raft_consensus.cc:2229] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:51:28.372969 27288 raft_consensus.cc:2258] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:28.373337 27288 ts_tablet_manager.cc:1905] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:28.373311 26848 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd4b70e3bfa4286aba4417638f18cb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:28 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:41738
I20260926 05:51:28.373788 27291 tablet_replica.cc:330] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927: stopping tablet replica
I20260926 05:51:28.373908 27291 raft_consensus.cc:2229] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:28.374007 27291 raft_consensus.cc:2258] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:28.374398 27291 ts_tablet_manager.cc:1905] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:28.375591 27288 ts_tablet_manager.cc:1918] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:28.375712 27288 log.cc:1196] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/8bd4b70e3bfa4286aba4417638f18cb3
I20260926 05:51:28.376057 27288 ts_tablet_manager.cc:1939] T 8bd4b70e3bfa4286aba4417638f18cb3 P 66f2951f40f64f098ca4b65e608159f6: Deleting consensus metadata
I20260926 05:51:28.376504 26696 catalog_manager.cc:4841] TS 66f2951f40f64f098ca4b65e608159f6 (127.24.255.2:43625): tablet 8bd4b70e3bfa4286aba4417638f18cb3 (table test-table [id=a011f1e1b1924741b830f7263ede6cf5]) successfully deleted
I20260926 05:51:28.376644 27291 ts_tablet_manager.cc:1918] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:28.376741 27291 log.cc:1196] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-0/wal/wals/8bd4b70e3bfa4286aba4417638f18cb3
I20260926 05:51:28.376971 27291 ts_tablet_manager.cc:1939] T 8bd4b70e3bfa4286aba4417638f18cb3 P 067e6de0cd3d4506b77e97217e56c927: Deleting consensus metadata
Sep 26 05:51:28 dist-test-slave-7fr8 krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401887, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.255.1@KRBTEST.COM
I20260926 05:51:28.380066 26696 catalog_manager.cc:4841] TS 067e6de0cd3d4506b77e97217e56c927 (127.24.255.1:41231): tablet 8bd4b70e3bfa4286aba4417638f18cb3 (table test-table [id=a011f1e1b1924741b830f7263ede6cf5]) successfully deleted
I20260926 05:51:28.382239 27289 tablet_replica.cc:330] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a: stopping tablet replica
I20260926 05:51:28.382378 27289 raft_consensus.cc:2229] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:28.382459 27289 raft_consensus.cc:2258] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:28.382911 27289 ts_tablet_manager.cc:1905] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:28.384227 27292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:28.373452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.24.255.1:41231)
0926 05:51:28.373741 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:28.373743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:28.373861 (+ 118us) server_negotiation.cc:407] Connection header received
0926 05:51:28.373885 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:28.373886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:28.373898 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:28.373953 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:28.374706 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.375414 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.376378 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.376657 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.376711 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:28.380428 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:28.380441 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:28.380441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:28.380450 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:28.382452 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:28.382936 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:28.382940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:28.382940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:28.383069 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:28.383275 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:28.383276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:28.383276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:28.383474 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:28.383586 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:28.384024 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 05:51:28.384117 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":92,"threads_started":1}
W20260926 05:51:28.384701 26900 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41752
I20260926 05:51:28.385447 27289 ts_tablet_manager.cc:1918] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:28.385505 27289 log.cc:1196] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/8bd4b70e3bfa4286aba4417638f18cb3
I20260926 05:51:28.385728 27289 ts_tablet_manager.cc:1939] T 8bd4b70e3bfa4286aba4417638f18cb3 P 7e61732d35474f2ba8a04da713e3a35a: Deleting consensus metadata
I20260926 05:51:28.386090 26695 catalog_manager.cc:4841] TS 7e61732d35474f2ba8a04da713e3a35a (127.24.255.3:44731): tablet 8bd4b70e3bfa4286aba4417638f18cb3 (table test-table [id=a011f1e1b1924741b830f7263ede6cf5]) successfully deleted
I20260926 05:51:28.393081 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:28.385693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36992 (local address 127.24.255.62:38291)
0926 05:51:28.385739 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:28.385741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:28.386071 (+ 330us) server_negotiation.cc:407] Connection header received
0926 05:51:28.386195 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:28.386196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:28.386209 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:28.386237 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:28.386858 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.387669 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.388679 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:28.388932 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:28.388991 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:28.390455 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:28.390457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:28.390457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:28.390466 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:28.392034 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:28.392355 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:28.392356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:28.392357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:28.392414 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:28.392572 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:28.392573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:28.392573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:28.392697 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:28.392792 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:28.392896 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 05:51:28.392986 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 05:51:28.393297 26711 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36992
I20260926 05:51:28.394749 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 26783
I20260926 05:51:28.404834 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 26923
I20260926 05:51:28.412092 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 27063
I20260926 05:51:28.416812 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 26679
2026-09-26T05:51:28Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3238 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:28 dist-test-slave-7fr8 krb5kdc[27298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:28 dist-test-slave-7fr8 krb5kdc[27298](info): set up 2 sockets
Sep 26 05:51:28 dist-test-slave-7fr8 krb5kdc[27298](info): commencing operation
krb5kdc: starting...
W20260926 05:51:29.414135 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:29 dist-test-slave-7fr8 krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:51:29.543267 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:40437
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:40437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:29.630690 27311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:29.630941 27311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:29.630970 27311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:29.632246 27311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:29.632292 27311 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 05:51:29.632303 27311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:29.632323 27311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:29.632334 27311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:29.632344 27311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:29.633955 27311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:40437
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:40437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:29.634231 27311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:29.634447 27311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:29.640326 27317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:29.640370 27319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:29.640341 27316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:29.640817 27311 server_base.cc:1034] running on GCE node
I20260926 05:51:29.641077 27311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:29.641397 27311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:29.642642 27311 hybrid_clock.cc:648] HybridClock initialized: now 1790401889642604 us; error 34 us; skew 500 ppm
Sep 26 05:51:29 dist-test-slave-7fr8 krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401889, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:29.650465 27311 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:51:29.651078 27311 webserver.cc:460] Webserver started at http://127.24.255.62:44531/ using document root <none> and password file <none>
I20260926 05:51:29.651335 27311 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:29.651394 27311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:29.651533 27311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:29.652688 27311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "772ae895127b4af99d992fe782666ed8"
format_stamp: "Formatted at 2026-09-26 05:51:29 on dist-test-slave-7fr8"
server_key: "97f8bc6996c6dbd94302a228d591740e"
server_key_iv: "f6ceac04d08f4174d3d30e6588dc9e47"
server_key_version: "encryptionkey@0"
I20260926 05:51:29.653069 27311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "772ae895127b4af99d992fe782666ed8"
format_stamp: "Formatted at 2026-09-26 05:51:29 on dist-test-slave-7fr8"
server_key: "97f8bc6996c6dbd94302a228d591740e"
server_key_iv: "f6ceac04d08f4174d3d30e6588dc9e47"
server_key_version: "encryptionkey@0"
I20260926 05:51:29.654791 27311 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 05:51:29.656330 27326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:29.656679 27311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:29.656795 27311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "772ae895127b4af99d992fe782666ed8"
format_stamp: "Formatted at 2026-09-26 05:51:29 on dist-test-slave-7fr8"
server_key: "97f8bc6996c6dbd94302a228d591740e"
server_key_iv: "f6ceac04d08f4174d3d30e6588dc9e47"
server_key_version: "encryptionkey@0"
I20260926 05:51:29.656886 27311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:29.670631 27311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:29.673616 27311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:29.673767 27311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:29.678642 27311 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:40437
I20260926 05:51:29.678718 27388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:40437 every 8 connection(s)
I20260926 05:51:29.679024 27311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:29.679360 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 27311
I20260926 05:51:29.679442 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:29.679605 25596 external_mini_cluster.cc:1428] Setting key bdd29643bcecf1f369288802ffbb5e24
I20260926 05:51:29.679703 27389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:29.683056 27389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8: Bootstrap starting.
I20260926 05:51:29.683697 27389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:29.683987 27389 log.cc:824] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:29.684748 27389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8: No bootstrap required, opened a new log
Sep 26 05:51:29 dist-test-slave-7fr8 krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:29.687291 27389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772ae895127b4af99d992fe782666ed8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 40437 } }
I20260926 05:51:29.687520 27389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:29.687562 27389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 772ae895127b4af99d992fe782666ed8, State: Initialized, Role: FOLLOWER
I20260926 05:51:29.687719 27389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772ae895127b4af99d992fe782666ed8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 40437 } }
I20260926 05:51:29.687811 27389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:29.687847 27389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:29.687896 27389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:29.688545 27389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772ae895127b4af99d992fe782666ed8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 40437 } }
I20260926 05:51:29.688651 27389 leader_election.cc:302] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 772ae895127b4af99d992fe782666ed8; no voters:
I20260926 05:51:29.688871 27389 leader_election.cc:288] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:29.689039 27394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:29.689332 27389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:29.689631 27394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [term 1 LEADER]: Becoming Leader. State: Replica: 772ae895127b4af99d992fe782666ed8, State: Running, Role: LEADER
I20260926 05:51:29.689870 27394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772ae895127b4af99d992fe782666ed8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 40437 } }
I20260926 05:51:29.690567 27396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 772ae895127b4af99d992fe782666ed8. Latest consensus state: current_term: 1 leader_uuid: "772ae895127b4af99d992fe782666ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772ae895127b4af99d992fe782666ed8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 40437 } } }
I20260926 05:51:29.690676 27396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:29.690903 27409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:29.691095 27395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "772ae895127b4af99d992fe782666ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772ae895127b4af99d992fe782666ed8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 40437 } } }
I20260926 05:51:29.691555 27395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:29.692003 27409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:29.692368 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:29.680115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.24.255.62:40437)
0926 05:51:29.680299 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:29.680309 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:29.680343 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:51:29.680866 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:29.680875 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:29.680940 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:29.681093 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:29.681644 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:29.683076 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:29.684219 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:29.684719 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:29.684820 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:29.687818 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:29.687822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:29.687823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:29.687835 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:29.690627 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:29.691101 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:29.691104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:29.691105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:29.691223 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:29.691439 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:29.691440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:29.691441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:29.691756 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:29.691900 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:29.692031 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 05:51:29.692172 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260926 05:51:29.693753 27409 catalog_manager.cc:1347] Generated new cluster ID: f0d711286c0c4a86aae093eb83ac2122
I20260926 05:51:29.693810 27409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:29.707753 27409 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:29.708346 27409 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:29.719605 27409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8: Generated new TSK 0
I20260926 05:51:29.719801 27409 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:51:29.752714 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:40437
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:29.842437 27415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:29.842664 27415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:29.842692 27415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:29.843992 27415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:29.844038 27415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:29.844059 27415 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 05:51:29.844070 27415 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:51:29.844086 27415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:29.845625 27415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.255.62:40437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:29.845854 27415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:29.846043 27415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:29.851375 27420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:29.851382 27423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:29.851382 27421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:29.851775 27415 server_base.cc:1034] running on GCE node
I20260926 05:51:29.852077 27415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:29.852375 27415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:29.853607 27415 hybrid_clock.cc:648] HybridClock initialized: now 1790401889853590 us; error 34 us; skew 500 ppm
Sep 26 05:51:29 dist-test-slave-7fr8 krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401889, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:29.856225 27415 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:51:29.856725 27415 webserver.cc:460] Webserver started at http://127.24.255.1:44215/ using document root <none> and password file <none>
I20260926 05:51:29.856936 27415 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:29.856985 27415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:29.857115 27415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:29.858253 27415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "681d141aa7c44470963e83a400f1362b"
format_stamp: "Formatted at 2026-09-26 05:51:29 on dist-test-slave-7fr8"
server_key: "ef7f795cdd16d867bfb728f4d26a2c88"
server_key_iv: "36e97344404cde1081d6211d8dd9d6b9"
server_key_version: "encryptionkey@0"
I20260926 05:51:29.858601 27415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "681d141aa7c44470963e83a400f1362b"
format_stamp: "Formatted at 2026-09-26 05:51:29 on dist-test-slave-7fr8"
server_key: "ef7f795cdd16d867bfb728f4d26a2c88"
server_key_iv: "36e97344404cde1081d6211d8dd9d6b9"
server_key_version: "encryptionkey@0"
I20260926 05:51:29.860139 27415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:51:29.861253 27430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:29.861490 27415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:29.861598 27415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "681d141aa7c44470963e83a400f1362b"
format_stamp: "Formatted at 2026-09-26 05:51:29 on dist-test-slave-7fr8"
server_key: "ef7f795cdd16d867bfb728f4d26a2c88"
server_key_iv: "36e97344404cde1081d6211d8dd9d6b9"
server_key_version: "encryptionkey@0"
I20260926 05:51:29.861672 27415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:29.876890 27415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:29.879099 27415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:29.879256 27415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:29.879797 27415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:29.879840 27415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:29.879873 27415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:29.879890 27415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:29.886731 27415 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:45807
I20260926 05:51:29.886864 27545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:45807 every 8 connection(s)
I20260926 05:51:29.887120 27415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:51:29 dist-test-slave-7fr8 krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401889, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:29.888119 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 27415
I20260926 05:51:29.888208 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:51:29.888445 25596 external_mini_cluster.cc:1428] Setting key c5555376f73cf24d959d02def84006a2
I20260926 05:51:29.891589 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:29.880413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:57157 (local address 127.24.255.62:40437)
0926 05:51:29.880478 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:29.880481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:29.880677 (+ 196us) server_negotiation.cc:407] Connection header received
0926 05:51:29.881361 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:29.881363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:29.881378 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:29.881440 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:29.882454 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:29.883267 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:29.884863 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:29.885073 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:29.885136 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:29.887958 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:29.887960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:29.887960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:29.887969 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:29.889470 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:29.890080 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:29.890083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:29.890083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:29.890256 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:29.890646 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:29.890647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:29.890647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:29.890817 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:29.890885 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:29.891352 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 05:51:29.891485 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:51:29.891817 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:29.880327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:40437 (local address 127.24.255.1:57157)
0926 05:51:29.880537 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:29.880579 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:29.881197 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:29.881523 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:29.881528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:29.882004 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:29.882324 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:29.882330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:29.883393 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:29.883393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:29.884756 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:29.884759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:29.885167 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:29.885168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:29.885334 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:29.886079 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:29.886118 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:29.887837 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:29.889542 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:29.889551 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:29.889554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:29.889876 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:29.890332 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:29.890333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:29.890333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:29.890567 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:29.890881 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:29.890890 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:29.891104 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 05:51:29.891327 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 05:51:29.891574 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":87,"threads_started":1}
I20260926 05:51:29.892493 27546 heartbeater.cc:347] Connected to a master server at 127.24.255.62:40437
I20260926 05:51:29.892652 27546 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:29.892920 27546 heartbeater.cc:511] Master 127.24.255.62:40437 requested a full tablet report, sending...
I20260926 05:51:29.893797 27342 ts_manager.cc:194] Registered new tserver with Master: 681d141aa7c44470963e83a400f1362b (127.24.255.1:45807)
I20260926 05:51:29.894608 27342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:57157
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
I20260926 05:51:29.899953 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:29.893747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:47925 (local address 127.24.255.62:40437)
0926 05:51:29.893786 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:29.893788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:29.893886 (+ 98us) server_negotiation.cc:407] Connection header received
0926 05:51:29.893989 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:29.893990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:29.894005 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:29.894036 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:29.894638 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:29.895177 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:29.896015 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:29.896216 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:29.896270 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:29.897444 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:29.897446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:29.897446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:29.897454 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:29.898830 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:29.899197 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:29.899199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:29.899199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:29.899254 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:29.899420 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:29.899420 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:29.899421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:29.899550 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:29.899614 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:29.899765 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 05:51:29.899850 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:51:29.899966 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:29.893578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:40437 (local address 127.24.255.1:47925)
0926 05:51:29.893794 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:29.893809 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:29.893921 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:29.894146 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:29.894147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:29.894428 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:29.894535 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:29.894537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:29.895321 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:29.895324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:29.895919 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:29.895921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:29.896280 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:29.896281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:29.896349 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:29.896951 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:29.896957 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:29.897349 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:29.898913 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:29.898914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:29.898915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:29.899125 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:29.899307 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:29.899308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:29.899309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:29.899370 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:29.899615 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:29.899616 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:29.899678 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 05:51:29.899737 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 05:51:29.899852 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:51:29.922571 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:40437
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:30.008061 27555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:30.008281 27555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:30.008307 27555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:30.009519 27555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:30.009563 27555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:30.009583 27555 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 05:51:30.009594 27555 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:51:30.009610 27555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:30.011188 27555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.255.62:40437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:30.011397 27555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:30.011651 27555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:30.016814 27560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:30.016812 27563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:30.016808 27561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:30.016987 27555 server_base.cc:1034] running on GCE node
I20260926 05:51:30.017256 27555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:30.017524 27555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:30.018676 27555 hybrid_clock.cc:648] HybridClock initialized: now 1790401890018619 us; error 74 us; skew 500 ppm
Sep 26 05:51:30 dist-test-slave-7fr8 krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401890, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:30.021224 27555 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:51:30.021674 27555 webserver.cc:460] Webserver started at http://127.24.255.2:37655/ using document root <none> and password file <none>
I20260926 05:51:30.021893 27555 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:30.021942 27555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:30.022038 27555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:30.023139 27555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "a2e789f59df248a58c449f4293a0b695"
format_stamp: "Formatted at 2026-09-26 05:51:30 on dist-test-slave-7fr8"
server_key: "ed97a7ce28c08980063ec37f7fa55958"
server_key_iv: "21821cefe588651bb32babe0f24062b3"
server_key_version: "encryptionkey@0"
I20260926 05:51:30.023487 27555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "a2e789f59df248a58c449f4293a0b695"
format_stamp: "Formatted at 2026-09-26 05:51:30 on dist-test-slave-7fr8"
server_key: "ed97a7ce28c08980063ec37f7fa55958"
server_key_iv: "21821cefe588651bb32babe0f24062b3"
server_key_version: "encryptionkey@0"
I20260926 05:51:30.024997 27555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:30.025933 27570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:30.026309 27555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:30.026424 27555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "a2e789f59df248a58c449f4293a0b695"
format_stamp: "Formatted at 2026-09-26 05:51:30 on dist-test-slave-7fr8"
server_key: "ed97a7ce28c08980063ec37f7fa55958"
server_key_iv: "21821cefe588651bb32babe0f24062b3"
server_key_version: "encryptionkey@0"
I20260926 05:51:30.026505 27555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:30.036861 27555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:30.039201 27555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:30.039323 27555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:30.039837 27555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:30.039889 27555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:30.039909 27555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:30.039921 27555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:30.046394 27555 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:43517
I20260926 05:51:30.046562 27685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:43517 every 8 connection(s)
I20260926 05:51:30.046800 27555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:51:30 dist-test-slave-7fr8 krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401890, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:30.048276 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 27555
I20260926 05:51:30.048419 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:51:30.048928 25596 external_mini_cluster.cc:1428] Setting key c7bd8de402eaa3aa2c14e955558f7372
I20260926 05:51:30.052002 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.040306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:45323 (local address 127.24.255.62:40437)
0926 05:51:30.040369 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.040371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.040629 (+ 258us) server_negotiation.cc:407] Connection header received
0926 05:51:30.041124 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.041125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.041136 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.041192 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:30.042043 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.042872 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.044476 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.044774 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.044874 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.048068 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:30.048070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:30.048070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:30.048079 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:30.049874 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:30.050386 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:30.050387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:30.050388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:30.050451 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:30.050664 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:30.050664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:30.050665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:30.050800 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:30.050869 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.051788 (+ 919us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.051914 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:51:30.052415 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.040278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:40437 (local address 127.24.255.2:45323)
0926 05:51:30.040502 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.040542 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.041030 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.041265 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.041270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.041619 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:30.041891 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.041897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.042994 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.042995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.044362 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.044365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.044962 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.044963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.045104 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.045803 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:30.045827 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:30.047944 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:30.049991 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:30.050000 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:30.050002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:30.050309 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:30.050506 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:30.050506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:30.050507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:30.050608 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:30.051043 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:30.051102 (+ 59us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:30.051465 (+ 363us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.051737 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.052040 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260926 05:51:30.053283 27686 heartbeater.cc:347] Connected to a master server at 127.24.255.62:40437
I20260926 05:51:30.053380 27686 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:30.053604 27686 heartbeater.cc:511] Master 127.24.255.62:40437 requested a full tablet report, sending...
I20260926 05:51:30.053992 27342 ts_manager.cc:194] Registered new tserver with Master: a2e789f59df248a58c449f4293a0b695 (127.24.255.2:43517)
I20260926 05:51:30.054558 27342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:45323
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
I20260926 05:51:30.062614 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.055367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:33203 (local address 127.24.255.62:40437)
0926 05:51:30.055408 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.055410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.055533 (+ 123us) server_negotiation.cc:407] Connection header received
0926 05:51:30.055624 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.055625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.055637 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.055699 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:30.056133 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.056687 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.057744 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.058211 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.058283 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.059468 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:30.059470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:30.059471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:30.059479 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:30.060847 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:30.061162 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:30.061163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:30.061164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:30.061249 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:30.061662 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:30.061665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:30.061665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:30.062152 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:30.062232 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.062411 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.062509 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:51:30.062625 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.055309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:40437 (local address 127.24.255.2:33203)
0926 05:51:30.055460 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.055471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.055534 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.055718 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.055719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.055946 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:30.056029 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.056031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.056787 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.056788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.057591 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.057594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.058322 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.058323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.058412 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.058890 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:30.058895 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:30.059325 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:30.060931 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:30.060932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:30.060932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:30.061100 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:30.061383 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:30.061384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:30.061384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:30.061454 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:30.062241 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:30.062242 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:30.062314 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.062373 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.062513 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:51:30.078186 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:40437
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:30.164373 27695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:30.164614 27695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:30.164642 27695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:30.166029 27695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:30.166126 27695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:30.166164 27695 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 05:51:30.166193 27695 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:51:30.166222 27695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:30.167876 27695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.255.62:40437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:30.168128 27695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:30.168371 27695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:30.173625 27701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:30.173629 27703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:30.173629 27700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:30.173875 27695 server_base.cc:1034] running on GCE node
I20260926 05:51:30.174417 27695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:30.174917 27695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:30.176118 27695 hybrid_clock.cc:648] HybridClock initialized: now 1790401890176090 us; error 46 us; skew 500 ppm
Sep 26 05:51:30 dist-test-slave-7fr8 krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401890, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:30.178789 27695 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:51:30.179257 27695 webserver.cc:460] Webserver started at http://127.24.255.3:42095/ using document root <none> and password file <none>
I20260926 05:51:30.179474 27695 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:30.179524 27695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:30.179653 27695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:30.180842 27695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "f835518719fe4b67b60cf9fc85052e74"
format_stamp: "Formatted at 2026-09-26 05:51:30 on dist-test-slave-7fr8"
server_key: "f5519595dcbbe738a672794f5bf30435"
server_key_iv: "f473bef977a2641c27293494f5d36654"
server_key_version: "encryptionkey@0"
I20260926 05:51:30.181191 27695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "f835518719fe4b67b60cf9fc85052e74"
format_stamp: "Formatted at 2026-09-26 05:51:30 on dist-test-slave-7fr8"
server_key: "f5519595dcbbe738a672794f5bf30435"
server_key_iv: "f473bef977a2641c27293494f5d36654"
server_key_version: "encryptionkey@0"
I20260926 05:51:30.182879 27695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:30.184006 27710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:30.184235 27695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:30.184393 27695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "f835518719fe4b67b60cf9fc85052e74"
format_stamp: "Formatted at 2026-09-26 05:51:30 on dist-test-slave-7fr8"
server_key: "f5519595dcbbe738a672794f5bf30435"
server_key_iv: "f473bef977a2641c27293494f5d36654"
server_key_version: "encryptionkey@0"
I20260926 05:51:30.184471 27695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:30.199395 27695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:30.201681 27695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:30.201833 27695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:30.202405 27695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:30.202453 27695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:30.202483 27695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:30.202507 27695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:51:30 dist-test-slave-7fr8 krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401890, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:30.209417 27695 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:43607
I20260926 05:51:30.209488 27825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:43607 every 8 connection(s)
I20260926 05:51:30.210145 27695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:51:30.213382 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 27695
I20260926 05:51:30.213472 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:51:30.213645 25596 external_mini_cluster.cc:1428] Setting key df7bbfbff691cd128c58536571d92e1f
I20260926 05:51:30.213640 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.202749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:45543 (local address 127.24.255.62:40437)
0926 05:51:30.202801 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.202804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.203087 (+ 283us) server_negotiation.cc:407] Connection header received
0926 05:51:30.203630 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.203632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.203646 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.203723 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:30.204498 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.205264 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.206588 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.206879 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.207017 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.209628 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:30.209629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:30.209629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:30.209637 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:30.211918 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:30.212394 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:30.212395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:30.212396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:30.212459 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:30.212782 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:30.212783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:30.212784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:30.212932 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:30.213016 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.213402 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.213551 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:51:30.213814 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.202751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:40437 (local address 127.24.255.3:45543)
0926 05:51:30.202912 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.202955 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.203531 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.203804 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.203807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.204106 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:30.204389 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.204396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.205359 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.205360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.206510 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.206513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.207033 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.207034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.207164 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.207706 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:30.207718 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:30.209526 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:30.212039 (+ 2513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:30.212045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:30.212047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:30.212293 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:30.212576 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:30.212577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:30.212577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:30.212693 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:30.213056 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:30.213067 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:30.213243 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.213345 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.213587 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260926 05:51:30.214356 27826 heartbeater.cc:347] Connected to a master server at 127.24.255.62:40437
I20260926 05:51:30.214452 27826 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:30.214628 27826 heartbeater.cc:511] Master 127.24.255.62:40437 requested a full tablet report, sending...
I20260926 05:51:30.215006 27343 ts_manager.cc:194] Registered new tserver with Master: f835518719fe4b67b60cf9fc85052e74 (127.24.255.3:43607)
I20260926 05:51:30.215483 27343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:45543
I20260926 05:51:30.220919 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.215387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:55001 (local address 127.24.255.62:40437)
0926 05:51:30.215443 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.215445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.215515 (+ 70us) server_negotiation.cc:407] Connection header received
0926 05:51:30.215570 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.215571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.215580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.215641 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:30.215991 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.216563 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.217343 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.217545 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.217609 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.218547 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:30.218549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:30.218549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:30.218555 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:30.219889 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:30.220221 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:30.220222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:30.220222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:30.220274 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:30.220436 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:30.220437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:30.220437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:30.220552 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:30.220608 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.220746 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.220821 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:51:30.220932 27832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.215330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:40437 (local address 127.24.255.3:55001)
0926 05:51:30.215455 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.215466 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.215527 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.215648 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.215649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.215850 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:30.215908 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.215910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.216669 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.216670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.217221 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.217223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.217613 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.217614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.217674 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.218136 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:30.218142 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:30.218463 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:30.219965 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:30.219966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:30.219966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:30.220155 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:30.220326 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:30.220327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:30.220327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:30.220386 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:30.220608 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:30.220609 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:30.220655 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.220714 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.220823 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260926 05:51:30.225646 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 05:51:30 dist-test-slave-7fr8 krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:51:30 dist-test-slave-7fr8 krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:30.248165 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.241024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.24.255.62:40437)
0926 05:51:30.241087 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.241089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.241099 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:30.241125 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.241126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.241135 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.241180 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:30.241736 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.242329 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.243054 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.243259 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.243317 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.245599 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:30.245602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:30.245602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:30.245610 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:30.247004 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:30.247356 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:30.247357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:30.247357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:30.247410 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:30.247571 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:30.247572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:30.247572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:30.247703 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:30.247807 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.247996 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.248089 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:51:30.250422 27343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:51:30.252003 27343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:30.259526 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.254795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:45807 (local address 127.0.0.1:33424)
0926 05:51:30.255478 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.255488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.255532 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.255713 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.255714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.255716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:30.255753 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.255755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.257055 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.257056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.258253 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.258254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.258938 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.258939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.259039 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.259055 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.259121 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.259416 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":90,"threads_started":1}
I20260926 05:51:30.259531 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.255317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.24.255.1:45807)
0926 05:51:30.255552 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.255556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.255564 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:30.255600 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.255602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.255616 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.255819 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:30.255867 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.256978 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.258336 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.258854 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.258928 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.259000 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.259145 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.259426 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
I20260926 05:51:30.259652 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.255112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36164 (local address 127.24.255.2:43517)
0926 05:51:30.255372 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.255375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.255646 (+ 271us) server_negotiation.cc:407] Connection header received
0926 05:51:30.255756 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.255758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.255773 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.255818 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:30.255980 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.256659 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.258276 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.259098 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.259175 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.259228 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.259471 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.259511 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":147,"threads_started":1}
I20260926 05:51:30.259804 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.255016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:43517 (local address 127.0.0.1:36164)
0926 05:51:30.255595 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.255602 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.255670 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.255876 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.255876 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.255878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:30.255918 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.255919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.256740 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.256741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.258195 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.258198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.259261 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.259262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.259327 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.259332 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.259377 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.259425 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":56,"threads_started":1}
I20260926 05:51:30.260275 27480 tablet_service.cc:1467] Processing CreateTablet for tablet 5c839fb07687476ab76770ededb10b73 (DEFAULT_TABLE table=test-table [id=d855587954344047932a68755d02d25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:30.260605 27480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c839fb07687476ab76770ededb10b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:30.260850 27620 tablet_service.cc:1467] Processing CreateTablet for tablet 5c839fb07687476ab76770ededb10b73 (DEFAULT_TABLE table=test-table [id=d855587954344047932a68755d02d25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:30.261207 27620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c839fb07687476ab76770ededb10b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:30.261620 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.254795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:43607 (local address 127.0.0.1:51420)
0926 05:51:30.255145 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.255190 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.255257 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.255599 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.255603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.255605 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:30.255682 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.255684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.258210 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.258211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.259341 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.259343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.261357 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.261358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.261434 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.261439 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.261500 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.261547 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":52,"threads_started":1}
I20260926 05:51:30.261664 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.255140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.24.255.3:43607)
0926 05:51:30.255359 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.255364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.255377 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:51:30.255417 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.255420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.255435 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.255511 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:30.255769 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.256965 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.259440 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.261253 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.261341 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.261400 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.261522 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.261573 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":81,"threads_started":1}
I20260926 05:51:30.262315 27760 tablet_service.cc:1467] Processing CreateTablet for tablet 5c839fb07687476ab76770ededb10b73 (DEFAULT_TABLE table=test-table [id=d855587954344047932a68755d02d25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:30.262676 27760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c839fb07687476ab76770ededb10b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:30.264567 27849 tablet_bootstrap.cc:492] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b: Bootstrap starting.
I20260926 05:51:30.264935 27850 tablet_bootstrap.cc:492] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695: Bootstrap starting.
I20260926 05:51:30.265478 27849 tablet_bootstrap.cc:654] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:30.265563 27850 tablet_bootstrap.cc:654] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:30.265808 27849 log.cc:824] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:30.265942 27850 log.cc:824] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:30.266286 27851 tablet_bootstrap.cc:492] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74: Bootstrap starting.
I20260926 05:51:30.266708 27850 tablet_bootstrap.cc:492] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695: No bootstrap required, opened a new log
I20260926 05:51:30.266702 27849 tablet_bootstrap.cc:492] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b: No bootstrap required, opened a new log
I20260926 05:51:30.266778 27849 ts_tablet_manager.cc:1397] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:51:30.266778 27850 ts_tablet_manager.cc:1397] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:51:30.267289 27851 tablet_bootstrap.cc:654] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:30.267567 27851 log.cc:824] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:30.268373 27851 tablet_bootstrap.cc:492] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74: No bootstrap required, opened a new log
I20260926 05:51:30.268440 27851 ts_tablet_manager.cc:1397] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:51:30.268813 27849 raft_consensus.cc:348] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } }
I20260926 05:51:30.268813 27850 raft_consensus.cc:348] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } }
I20260926 05:51:30.268996 27850 raft_consensus.cc:374] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:30.268996 27849 raft_consensus.cc:374] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:30.269032 27850 raft_consensus.cc:729] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e789f59df248a58c449f4293a0b695, State: Initialized, Role: FOLLOWER
I20260926 05:51:30.269032 27849 raft_consensus.cc:729] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681d141aa7c44470963e83a400f1362b, State: Initialized, Role: FOLLOWER
I20260926 05:51:30.269155 27850 consensus_queue.cc:260] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } }
I20260926 05:51:30.269155 27849 consensus_queue.cc:260] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } }
I20260926 05:51:30.269431 27849 ts_tablet_manager.cc:1428] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:51:30.269431 27850 ts_tablet_manager.cc:1428] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:51:30.269454 27546 heartbeater.cc:503] Master 127.24.255.62:40437 was elected leader, sending a full tablet report...
I20260926 05:51:30.269555 27686 heartbeater.cc:503] Master 127.24.255.62:40437 was elected leader, sending a full tablet report...
I20260926 05:51:30.270543 27851 raft_consensus.cc:348] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } }
I20260926 05:51:30.270709 27851 raft_consensus.cc:374] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:30.270748 27851 raft_consensus.cc:729] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835518719fe4b67b60cf9fc85052e74, State: Initialized, Role: FOLLOWER
I20260926 05:51:30.270886 27851 consensus_queue.cc:260] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } }
I20260926 05:51:30.271207 27851 ts_tablet_manager.cc:1428] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 05:51:30.271214 27826 heartbeater.cc:503] Master 127.24.255.62:40437 was elected leader, sending a full tablet report...
W20260926 05:51:30.297585 27687 tablet.cc:2369] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:51:30.388001 27547 tablet.cc:2369] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:51:30.460915 27827 tablet.cc:2369] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:51:30.494482 27855 raft_consensus.cc:482] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:30.494620 27855 raft_consensus.cc:504] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } }
I20260926 05:51:30.495011 27855 leader_election.cc:288] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f835518719fe4b67b60cf9fc85052e74 (127.24.255.3:43607), a2e789f59df248a58c449f4293a0b695 (127.24.255.2:43517)
I20260926 05:51:30.499848 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.495442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:43517 (local address 127.24.255.1:34215)
0926 05:51:30.495583 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.495596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.495661 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.495907 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.495909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.495923 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:30.496116 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.496121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.497147 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.497149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.498585 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.498589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.499454 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.499456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.499567 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.499574 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.499649 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.499740 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":185,"thread_start_us":70,"threads_started":1}
I20260926 05:51:30.499949 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.495572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:34215 (local address 127.24.255.2:43517)
0926 05:51:30.495614 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.495616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.495653 (+ 37us) server_negotiation.cc:407] Connection header received
0926 05:51:30.495751 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.495752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.495759 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.495807 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:30.496250 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.497026 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.498725 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.499299 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.499395 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.499406 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.499753 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.499841 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:51:30.499961 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.495571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:54779 (local address 127.24.255.3:43607)
0926 05:51:30.495613 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.495616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.495653 (+ 37us) server_negotiation.cc:407] Connection header received
0926 05:51:30.495727 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.495728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.495738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.495805 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:30.496305 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.497026 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.498725 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.499339 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.499404 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.499411 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.499775 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.499855 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:51:30.500095 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.495442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:43607 (local address 127.24.255.1:54779)
0926 05:51:30.495586 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.495596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.495661 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.495907 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.495909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.495924 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:30.496204 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.496206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.497147 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.497149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.498585 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.498589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.499453 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.499454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.499567 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.499575 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.499649 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.499735 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":69,"thread_start_us":67,"threads_started":1}
I20260926 05:51:30.500386 27640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c839fb07687476ab76770ededb10b73" candidate_uuid: "681d141aa7c44470963e83a400f1362b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e789f59df248a58c449f4293a0b695" is_pre_election: true
I20260926 05:51:30.500572 27640 raft_consensus.cc:2454] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681d141aa7c44470963e83a400f1362b in term 0.
I20260926 05:51:30.500831 27780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c839fb07687476ab76770ededb10b73" candidate_uuid: "681d141aa7c44470963e83a400f1362b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835518719fe4b67b60cf9fc85052e74" is_pre_election: true
I20260926 05:51:30.500872 27431 leader_election.cc:302] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 681d141aa7c44470963e83a400f1362b, a2e789f59df248a58c449f4293a0b695; no voters:
I20260926 05:51:30.500984 27780 raft_consensus.cc:2454] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681d141aa7c44470963e83a400f1362b in term 0.
I20260926 05:51:30.501000 27855 raft_consensus.cc:2781] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:30.501047 27855 raft_consensus.cc:482] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:30.501068 27855 raft_consensus.cc:3037] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:30.501799 27855 raft_consensus.cc:504] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } }
I20260926 05:51:30.501940 27855 leader_election.cc:288] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [CANDIDATE]: Term 1 election: Requested vote from peers f835518719fe4b67b60cf9fc85052e74 (127.24.255.3:43607), a2e789f59df248a58c449f4293a0b695 (127.24.255.2:43517)
I20260926 05:51:30.502141 27640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c839fb07687476ab76770ededb10b73" candidate_uuid: "681d141aa7c44470963e83a400f1362b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e789f59df248a58c449f4293a0b695"
I20260926 05:51:30.502239 27640 raft_consensus.cc:3037] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:30.502234 27780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c839fb07687476ab76770ededb10b73" candidate_uuid: "681d141aa7c44470963e83a400f1362b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835518719fe4b67b60cf9fc85052e74"
I20260926 05:51:30.502314 27780 raft_consensus.cc:3037] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:30.502945 27640 raft_consensus.cc:2454] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681d141aa7c44470963e83a400f1362b in term 1.
I20260926 05:51:30.503144 27780 raft_consensus.cc:2454] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681d141aa7c44470963e83a400f1362b in term 1.
I20260926 05:51:30.503273 27431 leader_election.cc:302] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 681d141aa7c44470963e83a400f1362b, a2e789f59df248a58c449f4293a0b695; no voters:
I20260926 05:51:30.503386 27855 raft_consensus.cc:2781] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:30.503508 27855 raft_consensus.cc:686] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [term 1 LEADER]: Becoming Leader. State: Replica: 681d141aa7c44470963e83a400f1362b, State: Running, Role: LEADER
I20260926 05:51:30.503603 27855 consensus_queue.cc:237] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } }
I20260926 05:51:30.504501 27343 catalog_manager.cc:5495] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b reported cstate change: term changed from 0 to 1, leader changed from <none> to 681d141aa7c44470963e83a400f1362b (127.24.255.1). New cstate: current_term: 1 leader_uuid: "681d141aa7c44470963e83a400f1362b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } health_report { overall_health: UNKNOWN } } }
Sep 26 05:51:30 dist-test-slave-7fr8 krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401889, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:30.574003 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.566974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.24.255.62:40437)
0926 05:51:30.567028 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.567030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.567108 (+ 78us) server_negotiation.cc:407] Connection header received
0926 05:51:30.567179 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.567180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.567189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.567239 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:30.567704 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.568209 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.568862 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.569040 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.569094 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.571476 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:30.571477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:30.571478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:30.571485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:30.572866 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:30.573249 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:30.573250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:30.573251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:30.573300 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:30.573503 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:30.573504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:30.573504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:30.573614 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:30.573669 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.573820 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.573906 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:51:30.574150 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.566881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:40437 (local address 127.0.0.1:43238)
0926 05:51:30.567025 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.567038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.567120 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.567252 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.567253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.567577 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:30.567624 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.567626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.568284 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.568285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.568793 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.568794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.569097 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.569098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.569146 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.569715 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:30.569720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:30.571362 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:30.572974 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:30.572986 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:30.572988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:30.573180 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:30.573354 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:30.573354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:30.573355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:30.573406 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:30.573673 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:30.573681 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:30.573731 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.573788 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.574015 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":70,"thread_start_us":75,"threads_started":1}
I20260926 05:51:30.576467 27343 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:43238:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 05:51:30.576684 27343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:30.578351 27480 tablet_service.cc:1467] Processing CreateTablet for tablet 4817fcb08bb74d8db8d53713f3efed8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc221c04a6f54b598c41e5a8544bb157]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:51:30.578398 27760 tablet_service.cc:1467] Processing CreateTablet for tablet 4817fcb08bb74d8db8d53713f3efed8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc221c04a6f54b598c41e5a8544bb157]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:51:30.578423 27620 tablet_service.cc:1467] Processing CreateTablet for tablet 4817fcb08bb74d8db8d53713f3efed8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc221c04a6f54b598c41e5a8544bb157]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:51:30.578541 27620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4817fcb08bb74d8db8d53713f3efed8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:30.578541 27480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4817fcb08bb74d8db8d53713f3efed8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:30.578553 27760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4817fcb08bb74d8db8d53713f3efed8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:30.579702 27851 tablet_bootstrap.cc:492] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74: Bootstrap starting.
I20260926 05:51:30.580008 27849 tablet_bootstrap.cc:492] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b: Bootstrap starting.
I20260926 05:51:30.580018 27850 tablet_bootstrap.cc:492] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695: Bootstrap starting.
I20260926 05:51:30.580242 27851 tablet_bootstrap.cc:654] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:30.580374 27849 tablet_bootstrap.cc:654] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:30.580395 27850 tablet_bootstrap.cc:654] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:30.581133 27849 tablet_bootstrap.cc:492] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b: No bootstrap required, opened a new log
I20260926 05:51:30.581141 27851 tablet_bootstrap.cc:492] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74: No bootstrap required, opened a new log
I20260926 05:51:30.581188 27851 ts_tablet_manager.cc:1397] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 05:51:30.581190 27849 ts_tablet_manager.cc:1397] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:30.581353 27851 raft_consensus.cc:348] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.581439 27851 raft_consensus.cc:374] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:30.581455 27851 raft_consensus.cc:729] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835518719fe4b67b60cf9fc85052e74, State: Initialized, Role: FOLLOWER
I20260926 05:51:30.581501 27851 consensus_queue.cc:260] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.581593 27851 ts_tablet_manager.cc:1428] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:30.581660 27857 tablet_replica.cc:439] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } }
I20260926 05:51:30.581768 27849 raft_consensus.cc:348] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.581748 27857 tablet_replica.cc:442] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:30.581845 27849 raft_consensus.cc:374] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:30.581866 27849 raft_consensus.cc:729] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681d141aa7c44470963e83a400f1362b, State: Initialized, Role: FOLLOWER
I20260926 05:51:30.581907 27849 consensus_queue.cc:260] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.581972 27850 tablet_bootstrap.cc:492] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695: No bootstrap required, opened a new log
I20260926 05:51:30.581988 27849 ts_tablet_manager.cc:1428] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:30.582021 27850 ts_tablet_manager.cc:1397] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:51:30.582198 27850 raft_consensus.cc:348] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.582378 27850 raft_consensus.cc:374] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:30.582300 27855 tablet_replica.cc:439] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } }
I20260926 05:51:30.582415 27850 raft_consensus.cc:729] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e789f59df248a58c449f4293a0b695, State: Initialized, Role: FOLLOWER
I20260926 05:51:30.582434 27855 tablet_replica.cc:442] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:30.582463 27850 consensus_queue.cc:260] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.582578 27850 ts_tablet_manager.cc:1428] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:30.582726 27856 tablet_replica.cc:439] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } }
I20260926 05:51:30.582818 27856 tablet_replica.cc:442] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:30.701354 27857 raft_consensus.cc:482] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:30.701485 27857 raft_consensus.cc:504] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.701970 27857 leader_election.cc:288] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 681d141aa7c44470963e83a400f1362b (127.24.255.1:45807), a2e789f59df248a58c449f4293a0b695 (127.24.255.2:43517)
I20260926 05:51:30.705749 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.702407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:43397 (local address 127.24.255.1:45807)
0926 05:51:30.702461 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.702463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.702523 (+ 60us) server_negotiation.cc:407] Connection header received
0926 05:51:30.702557 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.702558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.702568 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.702636 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:30.702925 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.703512 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.704816 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.705319 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.705424 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.705435 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.705616 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.705666 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 05:51:30.705798 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.702311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:45807 (local address 127.24.255.3:43397)
0926 05:51:30.702377 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.702391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.702445 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.702660 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.702661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.702664 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:30.702762 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.702766 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.703617 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.703618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.704681 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.704686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.705430 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.705431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.705534 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.705538 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.705588 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.705653 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 05:51:30.705847 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.702488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:49933 (local address 127.24.255.2:43517)
0926 05:51:30.702627 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.702630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.702693 (+ 63us) server_negotiation.cc:407] Connection header received
0926 05:51:30.702777 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.702778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.702789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.702846 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:30.703180 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.703798 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.704890 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.705342 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.705430 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.705444 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.705722 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.705782 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:51:30.706067 27871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.702414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:43517 (local address 127.24.255.3:49933)
0926 05:51:30.702607 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.702622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.702699 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.702922 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.702924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.702928 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:30.703048 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.703051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.703923 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.703924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.704741 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.704760 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.705511 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.705512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.705580 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.705588 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.705663 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.705718 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":100,"threads_started":1}
I20260926 05:51:30.706269 27500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4817fcb08bb74d8db8d53713f3efed8f" candidate_uuid: "f835518719fe4b67b60cf9fc85052e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681d141aa7c44470963e83a400f1362b" is_pre_election: true
I20260926 05:51:30.706400 27500 raft_consensus.cc:2454] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f835518719fe4b67b60cf9fc85052e74 in term 0.
I20260926 05:51:30.706358 27640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4817fcb08bb74d8db8d53713f3efed8f" candidate_uuid: "f835518719fe4b67b60cf9fc85052e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e789f59df248a58c449f4293a0b695" is_pre_election: true
I20260926 05:51:30.706454 27640 raft_consensus.cc:2454] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f835518719fe4b67b60cf9fc85052e74 in term 0.
I20260926 05:51:30.706677 27711 leader_election.cc:302] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2e789f59df248a58c449f4293a0b695, f835518719fe4b67b60cf9fc85052e74; no voters:
I20260926 05:51:30.706925 27857 raft_consensus.cc:2781] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:30.706986 27857 raft_consensus.cc:482] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:30.707001 27857 raft_consensus.cc:3037] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:30.707799 27857 raft_consensus.cc:504] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.707954 27857 leader_election.cc:288] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [CANDIDATE]: Term 1 election: Requested vote from peers 681d141aa7c44470963e83a400f1362b (127.24.255.1:45807), a2e789f59df248a58c449f4293a0b695 (127.24.255.2:43517)
I20260926 05:51:30.708204 27500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4817fcb08bb74d8db8d53713f3efed8f" candidate_uuid: "f835518719fe4b67b60cf9fc85052e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681d141aa7c44470963e83a400f1362b"
I20260926 05:51:30.708274 27500 raft_consensus.cc:3037] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:30.708236 27640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4817fcb08bb74d8db8d53713f3efed8f" candidate_uuid: "f835518719fe4b67b60cf9fc85052e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e789f59df248a58c449f4293a0b695"
I20260926 05:51:30.708354 27640 raft_consensus.cc:3037] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:30.708989 27640 raft_consensus.cc:2454] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f835518719fe4b67b60cf9fc85052e74 in term 1.
I20260926 05:51:30.709002 27500 raft_consensus.cc:2454] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f835518719fe4b67b60cf9fc85052e74 in term 1.
I20260926 05:51:30.709156 27711 leader_election.cc:302] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2e789f59df248a58c449f4293a0b695, f835518719fe4b67b60cf9fc85052e74; no voters:
I20260926 05:51:30.709259 27857 raft_consensus.cc:2781] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:30.709410 27857 raft_consensus.cc:686] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [term 1 LEADER]: Becoming Leader. State: Replica: f835518719fe4b67b60cf9fc85052e74, State: Running, Role: LEADER
I20260926 05:51:30.709502 27857 consensus_queue.cc:237] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.709944 27872 tablet_replica.cc:439] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74: TxnStatusTablet state changed. Reason: New leader f835518719fe4b67b60cf9fc85052e74. Latest consensus state: current_term: 1 leader_uuid: "f835518719fe4b67b60cf9fc85052e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } }
I20260926 05:51:30.710019 27872 tablet_replica.cc:442] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:51:30.710204 27874 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 05:51:30.710304 27874 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 05:51:30.710412 27343 catalog_manager.cc:5495] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 reported cstate change: term changed from 0 to 1, leader changed from <none> to f835518719fe4b67b60cf9fc85052e74 (127.24.255.3). New cstate: current_term: 1 leader_uuid: "f835518719fe4b67b60cf9fc85052e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } health_report { overall_health: HEALTHY } } }
I20260926 05:51:30.735255 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.732745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:43607 (local address 127.0.0.1:51428)
0926 05:51:30.732859 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.732871 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.732933 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.733103 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.733103 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.733104 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:51:30.733162 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.733165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.733766 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.733766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.734431 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.734434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.734749 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.734750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.734818 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.734834 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:51:30.735075 (+ 241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:51:30.735082 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.735120 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.735176 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 05:51:30.735248 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.732708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51428 (local address 127.24.255.3:43607)
0926 05:51:30.732761 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.732762 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.732923 (+ 161us) server_negotiation.cc:407] Connection header received
0926 05:51:30.733026 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.733027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.733044 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.733092 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:30.733218 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.733695 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.734505 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.734689 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.734744 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.734913 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:30.735019 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:30.735088 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.735134 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.735154 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 05:51:30.737169 27500 raft_consensus.cc:1261] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Refusing update from remote peer f835518719fe4b67b60cf9fc85052e74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:30.737171 27640 raft_consensus.cc:1261] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Refusing update from remote peer f835518719fe4b67b60cf9fc85052e74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:30.737514 27857 consensus_queue.cc:1035] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:30.737638 27872 consensus_queue.cc:1035] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:30.738804 27856 tablet_replica.cc:439] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695: TxnStatusTablet state changed. Reason: New leader f835518719fe4b67b60cf9fc85052e74. Latest consensus state: current_term: 1 leader_uuid: "f835518719fe4b67b60cf9fc85052e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } }
I20260926 05:51:30.738900 27856 tablet_replica.cc:442] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:30.739332 27855 tablet_replica.cc:439] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b: TxnStatusTablet state changed. Reason: New leader f835518719fe4b67b60cf9fc85052e74. Latest consensus state: current_term: 1 leader_uuid: "f835518719fe4b67b60cf9fc85052e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } }
I20260926 05:51:30.739430 27855 tablet_replica.cc:442] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:30.739730 27857 tablet_replica.cc:439] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f835518719fe4b67b60cf9fc85052e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } }
I20260926 05:51:30.739818 27857 tablet_replica.cc:442] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:51:30.739883 27877 mvcc.cc:205] Tried to move back new op lower bound from 7333486144456916992 to 7333486144346673152. Current Snapshot: MvccSnapshot[applied={T|T < 7333486144456916992}]
I20260926 05:51:30.740908 27857 tablet_replica.cc:439] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f835518719fe4b67b60cf9fc85052e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } }
I20260926 05:51:30.740994 27857 tablet_replica.cc:442] T 4817fcb08bb74d8db8d53713f3efed8f P f835518719fe4b67b60cf9fc85052e74: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:51:30.741233 27855 tablet_replica.cc:439] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f835518719fe4b67b60cf9fc85052e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } }
I20260926 05:51:30.741317 27855 tablet_replica.cc:442] T 4817fcb08bb74d8db8d53713f3efed8f P 681d141aa7c44470963e83a400f1362b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:30.741496 27856 tablet_replica.cc:439] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f835518719fe4b67b60cf9fc85052e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } } }
I20260926 05:51:30.741582 27856 tablet_replica.cc:442] T 4817fcb08bb74d8db8d53713f3efed8f P a2e789f59df248a58c449f4293a0b695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:30.742321 27878 mvcc.cc:205] Tried to move back new op lower bound from 7333486144456916992 to 7333486144346673152. Current Snapshot: MvccSnapshot[applied={T|T < 7333486144456916992}]
I20260926 05:51:30.746546 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.743058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.24.255.1:45807)
0926 05:51:30.743112 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.743113 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.743121 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:30.743153 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.743154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.743172 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.743257 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:30.743500 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.744278 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.745114 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.745419 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.745484 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.745691 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:30.745827 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:30.745912 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.746380 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.746430 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:51:30.753293 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.750325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:39181 (local address 127.24.255.3:43607)
0926 05:51:30.750372 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.750374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.750520 (+ 146us) server_negotiation.cc:407] Connection header received
0926 05:51:30.750583 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.750584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.750593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.750641 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:30.750844 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.751329 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.752521 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.752720 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.752804 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.753053 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:30.753085 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:30.753141 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.753210 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.753228 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:51:30.753371 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.750271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:43607 (local address 127.24.255.1:39181)
0926 05:51:30.750456 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.750467 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.750527 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.750657 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.750658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.750659 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:51:30.750732 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.750734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.751428 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.751429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.752405 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.752408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.752827 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.752828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.752904 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.752927 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:51:30.753157 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:51:30.753159 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.753207 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.753256 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
I20260926 05:51:30.755851 27640 raft_consensus.cc:1261] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Refusing update from remote peer 681d141aa7c44470963e83a400f1362b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:30.755970 27780 raft_consensus.cc:1261] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Refusing update from remote peer 681d141aa7c44470963e83a400f1362b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:30.756089 27855 consensus_queue.cc:1035] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:30.756209 27860 consensus_queue.cc:1035] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:30.757406 27879 mvcc.cc:205] Tried to move back new op lower bound from 7333486144534257664 to 7333486143503298560. Current Snapshot: MvccSnapshot[applied={T|T < 7333486144534257664}]
I20260926 05:51:30.765372 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.762271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:45807 (local address 127.24.255.3:46069)
0926 05:51:30.762416 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.762428 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.762485 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.762609 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.762610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.762611 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:51:30.762671 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.762673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.763578 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.763579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.764470 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.764472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.765014 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.765015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.765064 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.765080 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:51:30.765234 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:51:30.765236 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.765267 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.765301 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260926 05:51:30.765372 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.762464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:46069 (local address 127.24.255.1:45807)
0926 05:51:30.762507 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.762509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.762531 (+ 22us) server_negotiation.cc:407] Connection header received
0926 05:51:30.762551 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.762552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.762562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.762609 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:30.762710 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.763481 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.764603 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.764925 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.764982 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.765147 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:30.765181 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:30.765229 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.765270 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.765293 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:51:30.779276 27342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:30.779413 27342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:30.780278 27342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8: Sending DeleteTablet for 3 replicas of tablet 5c839fb07687476ab76770ededb10b73
I20260926 05:51:30.780606 27760 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c839fb07687476ab76770ededb10b73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:30 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:51420
I20260926 05:51:30.780705 27620 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c839fb07687476ab76770ededb10b73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:30 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:36164
I20260926 05:51:30.780994 27480 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c839fb07687476ab76770ededb10b73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:30 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:33424
I20260926 05:51:30.781262 27899 tablet_replica.cc:330] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b: stopping tablet replica
I20260926 05:51:30.781329 27342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:51:30.781493 27342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:30.781420 27899 raft_consensus.cc:2229] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:51:30.781797 27899 raft_consensus.cc:2258] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:30.782279 27899 ts_tablet_manager.cc:1905] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:30.782661 27760 tablet_service.cc:1467] Processing CreateTablet for tablet bebdd1af39e846299d6c6fe1ba3a3c35 (DEFAULT_TABLE table=test-table [id=6e4e83f4068249f0bf2783f55c3219cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:30.782792 27760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebdd1af39e846299d6c6fe1ba3a3c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:30.784157 27851 tablet_bootstrap.cc:492] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74: Bootstrap starting.
I20260926 05:51:30.784417 27899 ts_tablet_manager.cc:1918] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:30.784487 27851 tablet_bootstrap.cc:654] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:30.784585 27899 log.cc:1196] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/wal/wals/5c839fb07687476ab76770ededb10b73
I20260926 05:51:30.784904 27899 ts_tablet_manager.cc:1939] T 5c839fb07687476ab76770ededb10b73 P 681d141aa7c44470963e83a400f1362b: Deleting consensus metadata
I20260926 05:51:30.785157 27620 tablet_service.cc:1467] Processing CreateTablet for tablet bebdd1af39e846299d6c6fe1ba3a3c35 (DEFAULT_TABLE table=test-table [id=6e4e83f4068249f0bf2783f55c3219cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:30.785295 27620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebdd1af39e846299d6c6fe1ba3a3c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:30.786517 27898 tablet_replica.cc:330] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695: stopping tablet replica
I20260926 05:51:30.786695 27898 raft_consensus.cc:2229] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:30.786803 27898 raft_consensus.cc:2258] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:30.787009 27897 tablet_replica.cc:330] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74: stopping tablet replica
I20260926 05:51:30.787067 27850 tablet_bootstrap.cc:492] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695: Bootstrap starting.
I20260926 05:51:30.787192 27897 raft_consensus.cc:2229] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:30.787298 27897 raft_consensus.cc:2258] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:30.787488 27850 tablet_bootstrap.cc:654] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:30.787797 27897 ts_tablet_manager.cc:1905] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:30.788091 27480 tablet_service.cc:1467] Processing CreateTablet for tablet bebdd1af39e846299d6c6fe1ba3a3c35 (DEFAULT_TABLE table=test-table [id=6e4e83f4068249f0bf2783f55c3219cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:30.788220 27480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebdd1af39e846299d6c6fe1ba3a3c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:30.789516 27851 tablet_bootstrap.cc:492] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74: No bootstrap required, opened a new log
I20260926 05:51:30.789573 27851 ts_tablet_manager.cc:1397] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.000s
I20260926 05:51:30.789599 27849 tablet_bootstrap.cc:492] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b: Bootstrap starting.
I20260926 05:51:30.789733 27851 raft_consensus.cc:348] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.789821 27851 raft_consensus.cc:374] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:30.789846 27851 raft_consensus.cc:729] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835518719fe4b67b60cf9fc85052e74, State: Initialized, Role: FOLLOWER
I20260926 05:51:30.789898 27851 consensus_queue.cc:260] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.790019 27849 tablet_bootstrap.cc:654] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:30.790023 27851 ts_tablet_manager.cc:1428] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:30.790130 27897 ts_tablet_manager.cc:1918] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:30.790237 27897 log.cc:1196] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/5c839fb07687476ab76770ededb10b73
I20260926 05:51:30.790546 27327 catalog_manager.cc:4841] TS 681d141aa7c44470963e83a400f1362b (127.24.255.1:45807): tablet 5c839fb07687476ab76770ededb10b73 (table test-table [id=d855587954344047932a68755d02d25d]) successfully deleted
I20260926 05:51:30.791435 27898 ts_tablet_manager.cc:1905] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:30.791481 27897 ts_tablet_manager.cc:1939] T 5c839fb07687476ab76770ededb10b73 P f835518719fe4b67b60cf9fc85052e74: Deleting consensus metadata
I20260926 05:51:30.792078 27330 catalog_manager.cc:4841] TS f835518719fe4b67b60cf9fc85052e74 (127.24.255.3:43607): tablet 5c839fb07687476ab76770ededb10b73 (table test-table [id=d855587954344047932a68755d02d25d]) successfully deleted
I20260926 05:51:30.793586 27898 ts_tablet_manager.cc:1918] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:30.793671 27898 log.cc:1196] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/5c839fb07687476ab76770ededb10b73
I20260926 05:51:30.793906 27898 ts_tablet_manager.cc:1939] T 5c839fb07687476ab76770ededb10b73 P a2e789f59df248a58c449f4293a0b695: Deleting consensus metadata
I20260926 05:51:30.794373 27327 catalog_manager.cc:4841] TS a2e789f59df248a58c449f4293a0b695 (127.24.255.2:43517): tablet 5c839fb07687476ab76770ededb10b73 (table test-table [id=d855587954344047932a68755d02d25d]) successfully deleted
I20260926 05:51:30.795023 27850 tablet_bootstrap.cc:492] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695: No bootstrap required, opened a new log
I20260926 05:51:30.795074 27850 ts_tablet_manager.cc:1397] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260926 05:51:30.795104 27849 tablet_bootstrap.cc:492] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b: No bootstrap required, opened a new log
I20260926 05:51:30.795151 27849 ts_tablet_manager.cc:1397] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 05:51:30.795228 27850 raft_consensus.cc:348] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.795300 27850 raft_consensus.cc:374] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:30.795332 27850 raft_consensus.cc:729] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e789f59df248a58c449f4293a0b695, State: Initialized, Role: FOLLOWER
I20260926 05:51:30.795295 27849 raft_consensus.cc:348] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.795375 27849 raft_consensus.cc:374] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:30.795375 27850 consensus_queue.cc:260] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.795413 27849 raft_consensus.cc:729] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681d141aa7c44470963e83a400f1362b, State: Initialized, Role: FOLLOWER
I20260926 05:51:30.795470 27850 ts_tablet_manager.cc:1428] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:30.795460 27849 consensus_queue.cc:260] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.795544 27849 ts_tablet_manager.cc:1428] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:51:30.878197 27856 raft_consensus.cc:482] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:30.878304 27856 raft_consensus.cc:504] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.878654 27856 leader_election.cc:288] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 681d141aa7c44470963e83a400f1362b (127.24.255.1:45807), f835518719fe4b67b60cf9fc85052e74 (127.24.255.3:43607)
I20260926 05:51:30.882939 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.878844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:43607 (local address 127.24.255.2:44581)
0926 05:51:30.879026 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.879036 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.879126 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.879320 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.879320 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.879322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:30.879398 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.879400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.880737 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.880739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.881773 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.881775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.882619 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.882621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.882712 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.882717 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.882773 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.882823 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
I20260926 05:51:30.882970 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.878907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:44581 (local address 127.24.255.3:43607)
0926 05:51:30.878956 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.878958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.879095 (+ 137us) server_negotiation.cc:407] Connection header received
0926 05:51:30.879220 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.879221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.879231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.879315 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:30.879566 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.880610 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.881855 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.882528 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.882611 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.882624 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.882796 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.882881 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:51:30.883289 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.879069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:49531 (local address 127.24.255.1:45807)
0926 05:51:30.879116 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.879118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.879174 (+ 56us) server_negotiation.cc:407] Connection header received
0926 05:51:30.879262 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.879263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.879275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.879318 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:30.879537 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.880065 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.881635 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.882385 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.882470 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.882480 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.883187 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.883228 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:51:30.883262 27780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebdd1af39e846299d6c6fe1ba3a3c35" candidate_uuid: "a2e789f59df248a58c449f4293a0b695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835518719fe4b67b60cf9fc85052e74" is_pre_election: true
I20260926 05:51:30.883348 27780 raft_consensus.cc:2454] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e789f59df248a58c449f4293a0b695 in term 0.
I20260926 05:51:30.883522 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.878847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:45807 (local address 127.24.255.2:49531)
0926 05:51:30.879121 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.879130 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.879195 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.879359 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.879360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.879361 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:30.879410 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.879411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.880152 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.880153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.881522 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.881526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.882485 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.882486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.883088 (+ 602us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.883094 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.883389 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.883439 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":57,"threads_started":1}
I20260926 05:51:30.883543 27574 leader_election.cc:302] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2e789f59df248a58c449f4293a0b695, f835518719fe4b67b60cf9fc85052e74; no voters:
I20260926 05:51:30.883661 27856 raft_consensus.cc:2781] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:30.883697 27856 raft_consensus.cc:482] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:30.883714 27856 raft_consensus.cc:3037] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:30.883742 27500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebdd1af39e846299d6c6fe1ba3a3c35" candidate_uuid: "a2e789f59df248a58c449f4293a0b695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681d141aa7c44470963e83a400f1362b" is_pre_election: true
I20260926 05:51:30.883809 27500 raft_consensus.cc:2454] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e789f59df248a58c449f4293a0b695 in term 0.
I20260926 05:51:30.884677 27856 raft_consensus.cc:504] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.884835 27856 leader_election.cc:288] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [CANDIDATE]: Term 1 election: Requested vote from peers 681d141aa7c44470963e83a400f1362b (127.24.255.1:45807), f835518719fe4b67b60cf9fc85052e74 (127.24.255.3:43607)
I20260926 05:51:30.885007 27780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebdd1af39e846299d6c6fe1ba3a3c35" candidate_uuid: "a2e789f59df248a58c449f4293a0b695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835518719fe4b67b60cf9fc85052e74"
I20260926 05:51:30.884999 27500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebdd1af39e846299d6c6fe1ba3a3c35" candidate_uuid: "a2e789f59df248a58c449f4293a0b695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681d141aa7c44470963e83a400f1362b"
I20260926 05:51:30.885089 27780 raft_consensus.cc:3037] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:30.885102 27500 raft_consensus.cc:3037] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:30.885859 27780 raft_consensus.cc:2454] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e789f59df248a58c449f4293a0b695 in term 1.
I20260926 05:51:30.886023 27574 leader_election.cc:302] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2e789f59df248a58c449f4293a0b695, f835518719fe4b67b60cf9fc85052e74; no voters:
I20260926 05:51:30.886061 27500 raft_consensus.cc:2454] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e789f59df248a58c449f4293a0b695 in term 1.
I20260926 05:51:30.886152 27856 raft_consensus.cc:2781] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:30.886307 27856 raft_consensus.cc:686] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [term 1 LEADER]: Becoming Leader. State: Replica: a2e789f59df248a58c449f4293a0b695, State: Running, Role: LEADER
I20260926 05:51:30.886381 27856 consensus_queue.cc:237] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:30.887156 27342 catalog_manager.cc:5495] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e789f59df248a58c449f4293a0b695 (127.24.255.2). New cstate: current_term: 1 leader_uuid: "a2e789f59df248a58c449f4293a0b695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } health_report { overall_health: UNKNOWN } } }
I20260926 05:51:30.915831 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.913340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36178 (local address 127.24.255.2:43517)
0926 05:51:30.913416 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.913418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.913425 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:30.913457 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.913458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.913477 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.913604 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:30.913718 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.914327 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.915087 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.915286 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.915342 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.915504 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:30.915621 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:30.915685 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.915722 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.915747 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 05:51:30.921103 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.918592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:43607 (local address 127.24.255.2:42183)
0926 05:51:30.918764 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.918774 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.918833 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.919047 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.919048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.919049 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:51:30.919126 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.919127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.919745 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.919746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.920318 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.920319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.920692 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.920693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.920759 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.920773 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:51:30.920925 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:51:30.920927 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.920968 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.921012 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260926 05:51:30.921072 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.918765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:42183 (local address 127.24.255.3:43607)
0926 05:51:30.918826 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.918828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.918837 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:30.918970 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.918971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.918988 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.919037 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:30.919200 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.919676 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.920451 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.920626 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.920681 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.920847 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:30.920876 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:30.920917 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.920972 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.920989 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:51:30.923748 27500 raft_consensus.cc:1261] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Refusing update from remote peer a2e789f59df248a58c449f4293a0b695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:30.923758 27780 raft_consensus.cc:1261] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Refusing update from remote peer a2e789f59df248a58c449f4293a0b695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:30.924046 27856 consensus_queue.cc:1035] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:30.924194 27905 consensus_queue.cc:1035] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:30.931586 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.929163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:60001 (local address 127.24.255.2:43517)
0926 05:51:30.929204 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:30.929206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:30.929214 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:30.929236 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:30.929237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:30.929246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:30.929285 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:30.929415 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.929974 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.930861 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.931131 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.931199 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.931347 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:30.931382 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:30.931428 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:30.931484 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 05:51:30.931507 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:51:30.931620 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:30.929082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:43517 (local address 127.24.255.3:60001)
0926 05:51:30.929124 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:30.929137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:30.929177 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:30.929303 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:30.929304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:30.929306 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:51:30.929356 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.929358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.930085 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.930086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.930749 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:30.930752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:30.931218 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:30.931219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:30.931278 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:30.931285 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:51:30.931436 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:51:30.931438 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:51:30.931480 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 05:51:30.931534 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 05:51:30.931989 27878 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 05:51:32.459002 27343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:32.459105 27343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:32.459928 27343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8: Sending DeleteTablet for 3 replicas of tablet bebdd1af39e846299d6c6fe1ba3a3c35
I20260926 05:51:32.460151 27760 tablet_service.cc:1514] Processing DeleteTablet for tablet bebdd1af39e846299d6c6fe1ba3a3c35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:32 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:51420
I20260926 05:51:32.460184 27620 tablet_service.cc:1514] Processing DeleteTablet for tablet bebdd1af39e846299d6c6fe1ba3a3c35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:32 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:36164
I20260926 05:51:32.460193 27480 tablet_service.cc:1514] Processing DeleteTablet for tablet bebdd1af39e846299d6c6fe1ba3a3c35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:32 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:33424
I20260926 05:51:32.460464 27937 tablet_replica.cc:330] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74: stopping tablet replica
I20260926 05:51:32.460588 27939 tablet_replica.cc:330] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b: stopping tablet replica
I20260926 05:51:32.460637 27939 raft_consensus.cc:2229] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:32.460673 27939 raft_consensus.cc:2258] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:32.460728 27937 raft_consensus.cc:2229] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:32.460770 27937 raft_consensus.cc:2258] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:32.461052 27939 ts_tablet_manager.cc:1905] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:32.462687 27938 tablet_replica.cc:330] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695: stopping tablet replica
I20260926 05:51:32.462762 27938 raft_consensus.cc:2229] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:51:32.462767 27939 ts_tablet_manager.cc:1918] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:51:32.462810 27939 log.cc:1196] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/wal/wals/bebdd1af39e846299d6c6fe1ba3a3c35
I20260926 05:51:32.462872 27938 raft_consensus.cc:2258] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:32.463088 27938 ts_tablet_manager.cc:1905] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:32.463092 27937 ts_tablet_manager.cc:1905] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:32.464592 27938 ts_tablet_manager.cc:1918] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:51:32.464586 27937 ts_tablet_manager.cc:1918] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:51:32.464644 27938 log.cc:1196] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/bebdd1af39e846299d6c6fe1ba3a3c35
I20260926 05:51:32.464644 27937 log.cc:1196] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/bebdd1af39e846299d6c6fe1ba3a3c35
I20260926 05:51:32.464843 27937 ts_tablet_manager.cc:1939] T bebdd1af39e846299d6c6fe1ba3a3c35 P f835518719fe4b67b60cf9fc85052e74: Deleting consensus metadata
I20260926 05:51:32.464860 27938 ts_tablet_manager.cc:1939] T bebdd1af39e846299d6c6fe1ba3a3c35 P a2e789f59df248a58c449f4293a0b695: Deleting consensus metadata
I20260926 05:51:32.465095 27939 ts_tablet_manager.cc:1939] T bebdd1af39e846299d6c6fe1ba3a3c35 P 681d141aa7c44470963e83a400f1362b: Deleting consensus metadata
I20260926 05:51:32.465351 27327 catalog_manager.cc:4841] TS a2e789f59df248a58c449f4293a0b695 (127.24.255.2:43517): tablet bebdd1af39e846299d6c6fe1ba3a3c35 (table test-table [id=6e4e83f4068249f0bf2783f55c3219cc]) successfully deleted
I20260926 05:51:32.465418 27330 catalog_manager.cc:4841] TS f835518719fe4b67b60cf9fc85052e74 (127.24.255.3:43607): tablet bebdd1af39e846299d6c6fe1ba3a3c35 (table test-table [id=6e4e83f4068249f0bf2783f55c3219cc]) successfully deleted
I20260926 05:51:32.466177 27327 catalog_manager.cc:4841] TS 681d141aa7c44470963e83a400f1362b (127.24.255.1:45807): tablet bebdd1af39e846299d6c6fe1ba3a3c35 (table test-table [id=6e4e83f4068249f0bf2783f55c3219cc]) successfully deleted
Sep 26 05:51:32 dist-test-slave-7fr8 krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401892, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:51:32 dist-test-slave-7fr8 krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401892, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:32.483662 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:32.476346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.24.255.62:40437)
0926 05:51:32.476525 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:32.476527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:32.476551 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:51:32.476587 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:32.476588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:32.476598 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:32.476656 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:32.477303 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:32.477907 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:32.478820 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:32.479066 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:32.479134 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:32.481468 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:32.481470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:32.481470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:32.481479 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:32.482783 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:32.483090 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:32.483091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:32.483091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:32.483138 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:32.483265 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:32.483266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:32.483266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:32.483375 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:32.483446 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:32.483524 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 05:51:32.483586 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":104,"threads_started":1}
I20260926 05:51:32.484964 27343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:51:32.485117 27343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:32.486449 27760 tablet_service.cc:1467] Processing CreateTablet for tablet eb6cbd395a2e47ebbbcfb0c8b780f1b6 (DEFAULT_TABLE table=test-table [id=e4d9db8ecf1f49fb9ef1cb81e8a0e567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:32.486449 27480 tablet_service.cc:1467] Processing CreateTablet for tablet eb6cbd395a2e47ebbbcfb0c8b780f1b6 (DEFAULT_TABLE table=test-table [id=e4d9db8ecf1f49fb9ef1cb81e8a0e567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:32.486470 27620 tablet_service.cc:1467] Processing CreateTablet for tablet eb6cbd395a2e47ebbbcfb0c8b780f1b6 (DEFAULT_TABLE table=test-table [id=e4d9db8ecf1f49fb9ef1cb81e8a0e567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:32.486588 27480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6cbd395a2e47ebbbcfb0c8b780f1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:32.486585 27760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6cbd395a2e47ebbbcfb0c8b780f1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:32.486605 27620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6cbd395a2e47ebbbcfb0c8b780f1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:32.487644 27948 tablet_bootstrap.cc:492] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695: Bootstrap starting.
I20260926 05:51:32.488083 27950 tablet_bootstrap.cc:492] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b: Bootstrap starting.
I20260926 05:51:32.488140 27948 tablet_bootstrap.cc:654] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:32.488425 27950 tablet_bootstrap.cc:654] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:32.488658 27948 tablet_bootstrap.cc:492] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695: No bootstrap required, opened a new log
I20260926 05:51:32.488703 27948 ts_tablet_manager.cc:1397] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:32.488711 27949 tablet_bootstrap.cc:492] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74: Bootstrap starting.
I20260926 05:51:32.488817 27948 raft_consensus.cc:348] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.488895 27948 raft_consensus.cc:374] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:32.488940 27950 tablet_bootstrap.cc:492] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b: No bootstrap required, opened a new log
I20260926 05:51:32.488965 27948 raft_consensus.cc:729] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e789f59df248a58c449f4293a0b695, State: Initialized, Role: FOLLOWER
I20260926 05:51:32.488991 27950 ts_tablet_manager.cc:1397] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:32.489041 27949 tablet_bootstrap.cc:654] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:32.489024 27948 consensus_queue.cc:260] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.489131 27948 ts_tablet_manager.cc:1428] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:32.489347 27950 raft_consensus.cc:348] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.489426 27950 raft_consensus.cc:374] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:32.489440 27950 raft_consensus.cc:729] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681d141aa7c44470963e83a400f1362b, State: Initialized, Role: FOLLOWER
I20260926 05:51:32.489480 27950 consensus_queue.cc:260] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.489635 27950 ts_tablet_manager.cc:1428] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:32.489749 27949 tablet_bootstrap.cc:492] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74: No bootstrap required, opened a new log
I20260926 05:51:32.489795 27949 ts_tablet_manager.cc:1397] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:32.490011 27949 raft_consensus.cc:348] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.490087 27949 raft_consensus.cc:374] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:32.490127 27949 raft_consensus.cc:729] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835518719fe4b67b60cf9fc85052e74, State: Initialized, Role: FOLLOWER
I20260926 05:51:32.490166 27949 consensus_queue.cc:260] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.490365 27949 ts_tablet_manager.cc:1428] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:32.557852 27954 raft_consensus.cc:482] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:32.557960 27954 raft_consensus.cc:504] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.558153 27954 leader_election.cc:288] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e789f59df248a58c449f4293a0b695 (127.24.255.2:43517), f835518719fe4b67b60cf9fc85052e74 (127.24.255.3:43607)
I20260926 05:51:32.558394 27780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6cbd395a2e47ebbbcfb0c8b780f1b6" candidate_uuid: "681d141aa7c44470963e83a400f1362b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835518719fe4b67b60cf9fc85052e74" is_pre_election: true
I20260926 05:51:32.558478 27780 raft_consensus.cc:2454] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681d141aa7c44470963e83a400f1362b in term 0.
I20260926 05:51:32.558408 27640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6cbd395a2e47ebbbcfb0c8b780f1b6" candidate_uuid: "681d141aa7c44470963e83a400f1362b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e789f59df248a58c449f4293a0b695" is_pre_election: true
I20260926 05:51:32.558523 27640 raft_consensus.cc:2454] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681d141aa7c44470963e83a400f1362b in term 0.
I20260926 05:51:32.558637 27434 leader_election.cc:302] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 681d141aa7c44470963e83a400f1362b, f835518719fe4b67b60cf9fc85052e74; no voters:
I20260926 05:51:32.558720 27954 raft_consensus.cc:2781] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:32.558753 27954 raft_consensus.cc:482] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:32.558777 27954 raft_consensus.cc:3037] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:32.559697 27954 raft_consensus.cc:504] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.559847 27954 leader_election.cc:288] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [CANDIDATE]: Term 1 election: Requested vote from peers a2e789f59df248a58c449f4293a0b695 (127.24.255.2:43517), f835518719fe4b67b60cf9fc85052e74 (127.24.255.3:43607)
I20260926 05:51:32.559995 27640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6cbd395a2e47ebbbcfb0c8b780f1b6" candidate_uuid: "681d141aa7c44470963e83a400f1362b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e789f59df248a58c449f4293a0b695"
I20260926 05:51:32.560006 27780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6cbd395a2e47ebbbcfb0c8b780f1b6" candidate_uuid: "681d141aa7c44470963e83a400f1362b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835518719fe4b67b60cf9fc85052e74"
I20260926 05:51:32.560065 27640 raft_consensus.cc:3037] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:32.560073 27780 raft_consensus.cc:3037] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:32.560670 27640 raft_consensus.cc:2454] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681d141aa7c44470963e83a400f1362b in term 1.
I20260926 05:51:32.560792 27780 raft_consensus.cc:2454] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681d141aa7c44470963e83a400f1362b in term 1.
I20260926 05:51:32.560868 27431 leader_election.cc:302] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 681d141aa7c44470963e83a400f1362b, a2e789f59df248a58c449f4293a0b695; no voters:
I20260926 05:51:32.560961 27954 raft_consensus.cc:2781] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:32.561108 27954 raft_consensus.cc:686] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [term 1 LEADER]: Becoming Leader. State: Replica: 681d141aa7c44470963e83a400f1362b, State: Running, Role: LEADER
I20260926 05:51:32.561156 27954 consensus_queue.cc:237] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.561693 27342 catalog_manager.cc:5495] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b reported cstate change: term changed from 0 to 1, leader changed from <none> to 681d141aa7c44470963e83a400f1362b (127.24.255.1). New cstate: current_term: 1 leader_uuid: "681d141aa7c44470963e83a400f1362b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } health_report { overall_health: UNKNOWN } } }
I20260926 05:51:32.586175 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:32.584032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.24.255.1:45807)
0926 05:51:32.584169 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:32.584171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:32.584179 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:32.584198 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:32.584199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:32.584208 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:32.584260 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:32.584366 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:32.584842 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:32.585586 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:32.585797 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:32.585858 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:32.585949 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:32.585978 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:32.586009 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:32.586063 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 05:51:32.586078 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260926 05:51:32.588590 27780 raft_consensus.cc:1261] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Refusing update from remote peer 681d141aa7c44470963e83a400f1362b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:32.588593 27640 raft_consensus.cc:1261] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Refusing update from remote peer 681d141aa7c44470963e83a400f1362b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:32.588799 27954 consensus_queue.cc:1035] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:32.588917 27955 consensus_queue.cc:1035] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:32.599143 27342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:32.599246 27342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:32.599954 27342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8: Sending DeleteTablet for 3 replicas of tablet eb6cbd395a2e47ebbbcfb0c8b780f1b6
I20260926 05:51:32.600172 27760 tablet_service.cc:1514] Processing DeleteTablet for tablet eb6cbd395a2e47ebbbcfb0c8b780f1b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:32 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:51420
I20260926 05:51:32.600232 27620 tablet_service.cc:1514] Processing DeleteTablet for tablet eb6cbd395a2e47ebbbcfb0c8b780f1b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:32 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:36164
I20260926 05:51:32.600312 27938 tablet_replica.cc:330] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695: stopping tablet replica
I20260926 05:51:32.600314 27937 tablet_replica.cc:330] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74: stopping tablet replica
I20260926 05:51:32.600361 27938 raft_consensus.cc:2229] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:32.600400 27938 raft_consensus.cc:2258] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:32.600423 27937 raft_consensus.cc:2229] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:32.600478 27937 raft_consensus.cc:2258] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:32.600540 27480 tablet_service.cc:1514] Processing DeleteTablet for tablet eb6cbd395a2e47ebbbcfb0c8b780f1b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:32 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:33424
I20260926 05:51:32.600554 27342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 05:51:32.600620 27939 tablet_replica.cc:330] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b: stopping tablet replica
I20260926 05:51:32.600672 27939 raft_consensus.cc:2229] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [term 1 LEADER]: Raft consensus shutting down.
W20260926 05:51:32.600683 27342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:32.600725 27939 raft_consensus.cc:2258] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:32.600845 27938 ts_tablet_manager.cc:1905] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:32.600953 27939 ts_tablet_manager.cc:1905] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:32.601842 27760 tablet_service.cc:1467] Processing CreateTablet for tablet 802a71a88a4547fbbb009bfdbfe43daf (DEFAULT_TABLE table=test-table [id=f122c63e41a946a68777ed0e68718ea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:32.601971 27760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802a71a88a4547fbbb009bfdbfe43daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:32.602196 27620 tablet_service.cc:1467] Processing CreateTablet for tablet 802a71a88a4547fbbb009bfdbfe43daf (DEFAULT_TABLE table=test-table [id=f122c63e41a946a68777ed0e68718ea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:32.602288 27620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802a71a88a4547fbbb009bfdbfe43daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:32.602418 27480 tablet_service.cc:1467] Processing CreateTablet for tablet 802a71a88a4547fbbb009bfdbfe43daf (DEFAULT_TABLE table=test-table [id=f122c63e41a946a68777ed0e68718ea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:32.602593 27480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802a71a88a4547fbbb009bfdbfe43daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:32.602597 27938 ts_tablet_manager.cc:1918] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:32.602597 27939 ts_tablet_manager.cc:1918] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:32.602646 27938 log.cc:1196] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/eb6cbd395a2e47ebbbcfb0c8b780f1b6
I20260926 05:51:32.602658 27939 log.cc:1196] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-0/wal/wals/eb6cbd395a2e47ebbbcfb0c8b780f1b6
I20260926 05:51:32.602844 27939 ts_tablet_manager.cc:1939] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P 681d141aa7c44470963e83a400f1362b: Deleting consensus metadata
I20260926 05:51:32.602844 27938 ts_tablet_manager.cc:1939] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P a2e789f59df248a58c449f4293a0b695: Deleting consensus metadata
I20260926 05:51:32.603125 27327 catalog_manager.cc:4841] TS a2e789f59df248a58c449f4293a0b695 (127.24.255.2:43517): tablet eb6cbd395a2e47ebbbcfb0c8b780f1b6 (table test-table [id=e4d9db8ecf1f49fb9ef1cb81e8a0e567]) successfully deleted
I20260926 05:51:32.603264 27937 ts_tablet_manager.cc:1905] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:32.603510 27949 tablet_bootstrap.cc:492] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74: Bootstrap starting.
I20260926 05:51:32.603762 27950 tablet_bootstrap.cc:492] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b: Bootstrap starting.
I20260926 05:51:32.603813 27949 tablet_bootstrap.cc:654] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:32.604072 27950 tablet_bootstrap.cc:654] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:32.604334 27949 tablet_bootstrap.cc:492] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74: No bootstrap required, opened a new log
I20260926 05:51:32.604382 27949 ts_tablet_manager.cc:1397] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:32.604496 27949 raft_consensus.cc:348] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.604568 27949 raft_consensus.cc:374] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:32.604584 27937 ts_tablet_manager.cc:1918] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:32.604586 27949 raft_consensus.cc:729] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f835518719fe4b67b60cf9fc85052e74, State: Initialized, Role: FOLLOWER
I20260926 05:51:32.604645 27950 tablet_bootstrap.cc:492] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b: No bootstrap required, opened a new log
I20260926 05:51:32.604662 27937 log.cc:1196] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/eb6cbd395a2e47ebbbcfb0c8b780f1b6
I20260926 05:51:32.604681 27950 ts_tablet_manager.cc:1397] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:32.604677 27949 consensus_queue.cc:260] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.604753 27949 ts_tablet_manager.cc:1428] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:32.604780 27950 raft_consensus.cc:348] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.604839 27950 raft_consensus.cc:374] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:32.604853 27937 ts_tablet_manager.cc:1939] T eb6cbd395a2e47ebbbcfb0c8b780f1b6 P f835518719fe4b67b60cf9fc85052e74: Deleting consensus metadata
I20260926 05:51:32.604857 27950 raft_consensus.cc:729] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681d141aa7c44470963e83a400f1362b, State: Initialized, Role: FOLLOWER
I20260926 05:51:32.604915 27950 consensus_queue.cc:260] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.605000 27950 ts_tablet_manager.cc:1428] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:32.605139 27330 catalog_manager.cc:4841] TS f835518719fe4b67b60cf9fc85052e74 (127.24.255.3:43607): tablet eb6cbd395a2e47ebbbcfb0c8b780f1b6 (table test-table [id=e4d9db8ecf1f49fb9ef1cb81e8a0e567]) successfully deleted
I20260926 05:51:32.605418 27327 catalog_manager.cc:4841] TS 681d141aa7c44470963e83a400f1362b (127.24.255.1:45807): tablet eb6cbd395a2e47ebbbcfb0c8b780f1b6 (table test-table [id=e4d9db8ecf1f49fb9ef1cb81e8a0e567]) successfully deleted
I20260926 05:51:32.605612 27948 tablet_bootstrap.cc:492] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695: Bootstrap starting.
I20260926 05:51:32.605945 27948 tablet_bootstrap.cc:654] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:32.606437 27948 tablet_bootstrap.cc:492] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695: No bootstrap required, opened a new log
I20260926 05:51:32.606479 27948 ts_tablet_manager.cc:1397] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:32.606600 27948 raft_consensus.cc:348] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.606655 27948 raft_consensus.cc:374] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:32.606673 27948 raft_consensus.cc:729] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e789f59df248a58c449f4293a0b695, State: Initialized, Role: FOLLOWER
I20260926 05:51:32.606714 27948 consensus_queue.cc:260] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.606805 27948 ts_tablet_manager.cc:1428] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:32.784603 27963 raft_consensus.cc:482] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:32.784719 27963 raft_consensus.cc:504] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.784893 27963 leader_election.cc:288] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e789f59df248a58c449f4293a0b695 (127.24.255.2:43517), f835518719fe4b67b60cf9fc85052e74 (127.24.255.3:43607)
I20260926 05:51:32.785130 27780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802a71a88a4547fbbb009bfdbfe43daf" candidate_uuid: "681d141aa7c44470963e83a400f1362b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835518719fe4b67b60cf9fc85052e74" is_pre_election: true
I20260926 05:51:32.785123 27640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802a71a88a4547fbbb009bfdbfe43daf" candidate_uuid: "681d141aa7c44470963e83a400f1362b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e789f59df248a58c449f4293a0b695" is_pre_election: true
I20260926 05:51:32.785213 27780 raft_consensus.cc:2454] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681d141aa7c44470963e83a400f1362b in term 0.
I20260926 05:51:32.785226 27640 raft_consensus.cc:2454] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681d141aa7c44470963e83a400f1362b in term 0.
I20260926 05:51:32.785358 27434 leader_election.cc:302] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 681d141aa7c44470963e83a400f1362b, f835518719fe4b67b60cf9fc85052e74; no voters:
I20260926 05:51:32.785476 27963 raft_consensus.cc:2781] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:32.785524 27963 raft_consensus.cc:482] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:32.785538 27963 raft_consensus.cc:3037] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:32.786391 27963 raft_consensus.cc:504] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.786536 27963 leader_election.cc:288] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [CANDIDATE]: Term 1 election: Requested vote from peers a2e789f59df248a58c449f4293a0b695 (127.24.255.2:43517), f835518719fe4b67b60cf9fc85052e74 (127.24.255.3:43607)
I20260926 05:51:32.786674 27640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802a71a88a4547fbbb009bfdbfe43daf" candidate_uuid: "681d141aa7c44470963e83a400f1362b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e789f59df248a58c449f4293a0b695"
I20260926 05:51:32.786684 27780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802a71a88a4547fbbb009bfdbfe43daf" candidate_uuid: "681d141aa7c44470963e83a400f1362b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f835518719fe4b67b60cf9fc85052e74"
I20260926 05:51:32.786753 27780 raft_consensus.cc:3037] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:32.786763 27640 raft_consensus.cc:3037] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:32.787353 27780 raft_consensus.cc:2454] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681d141aa7c44470963e83a400f1362b in term 1.
I20260926 05:51:32.787530 27434 leader_election.cc:302] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 681d141aa7c44470963e83a400f1362b, f835518719fe4b67b60cf9fc85052e74; no voters:
I20260926 05:51:32.787603 27640 raft_consensus.cc:2454] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681d141aa7c44470963e83a400f1362b in term 1.
I20260926 05:51:32.787618 27963 raft_consensus.cc:2781] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:32.787662 27963 raft_consensus.cc:686] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [term 1 LEADER]: Becoming Leader. State: Replica: 681d141aa7c44470963e83a400f1362b, State: Running, Role: LEADER
I20260926 05:51:32.787708 27963 consensus_queue.cc:237] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } }
I20260926 05:51:32.788344 27343 catalog_manager.cc:5495] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b reported cstate change: term changed from 0 to 1, leader changed from <none> to 681d141aa7c44470963e83a400f1362b (127.24.255.1). New cstate: current_term: 1 leader_uuid: "681d141aa7c44470963e83a400f1362b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681d141aa7c44470963e83a400f1362b" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 } health_report { overall_health: UNKNOWN } } }
I20260926 05:51:32.836603 27640 raft_consensus.cc:1261] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Refusing update from remote peer 681d141aa7c44470963e83a400f1362b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:32.836637 27780 raft_consensus.cc:1261] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Refusing update from remote peer 681d141aa7c44470963e83a400f1362b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:32.836817 27963 consensus_queue.cc:1035] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f835518719fe4b67b60cf9fc85052e74" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:32.836962 27955 consensus_queue.cc:1035] T 802a71a88a4547fbbb009bfdbfe43daf P 681d141aa7c44470963e83a400f1362b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e789f59df248a58c449f4293a0b695" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 05:51:32.841501 27919 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 05:51:34.358835 27343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:34.358938 27343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:34.359822 27343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 772ae895127b4af99d992fe782666ed8: Sending DeleteTablet for 3 replicas of tablet 802a71a88a4547fbbb009bfdbfe43daf
I20260926 05:51:34.360083 27620 tablet_service.cc:1514] Processing DeleteTablet for tablet 802a71a88a4547fbbb009bfdbfe43daf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:34 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:36164
I20260926 05:51:34.360088 27480 tablet_service.cc:1514] Processing DeleteTablet for tablet 802a71a88a4547fbbb009bfdbfe43daf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:34 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:33424
I20260926 05:51:34.360272 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 27415
I20260926 05:51:34.360320 27760 tablet_service.cc:1514] Processing DeleteTablet for tablet 802a71a88a4547fbbb009bfdbfe43daf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:34 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:51420
I20260926 05:51:34.362488 27993 tablet_replica.cc:330] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74: stopping tablet replica
I20260926 05:51:34.362565 27993 raft_consensus.cc:2229] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:34.362615 27993 raft_consensus.cc:2258] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:34.362728 27991 tablet_replica.cc:330] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695: stopping tablet replica
I20260926 05:51:34.362790 27991 raft_consensus.cc:2229] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:34.362864 27991 raft_consensus.cc:2258] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:34.362879 27993 ts_tablet_manager.cc:1905] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:34.363097 27991 ts_tablet_manager.cc:1905] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:34.365058 27991 ts_tablet_manager.cc:1918] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:51:34.365135 27991 log.cc:1196] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/802a71a88a4547fbbb009bfdbfe43daf
I20260926 05:51:34.365320 27993 ts_tablet_manager.cc:1918] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:51:34.365365 27991 ts_tablet_manager.cc:1939] T 802a71a88a4547fbbb009bfdbfe43daf P a2e789f59df248a58c449f4293a0b695: Deleting consensus metadata
I20260926 05:51:34.365388 27993 log.cc:1196] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/802a71a88a4547fbbb009bfdbfe43daf
I20260926 05:51:34.365576 27993 ts_tablet_manager.cc:1939] T 802a71a88a4547fbbb009bfdbfe43daf P f835518719fe4b67b60cf9fc85052e74: Deleting consensus metadata
I20260926 05:51:34.365841 27327 catalog_manager.cc:4841] TS a2e789f59df248a58c449f4293a0b695 (127.24.255.2:43517): tablet 802a71a88a4547fbbb009bfdbfe43daf (table test-table [id=f122c63e41a946a68777ed0e68718ea7]) successfully deleted
I20260926 05:51:34.365890 27330 catalog_manager.cc:4841] TS f835518719fe4b67b60cf9fc85052e74 (127.24.255.3:43607): tablet 802a71a88a4547fbbb009bfdbfe43daf (table test-table [id=f122c63e41a946a68777ed0e68718ea7]) successfully deleted
W20260926 05:51:34.367594 27327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:45807 (error 108)
I20260926 05:51:34.367867 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 27555
I20260926 05:51:34.368561 27995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:34.367953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:45807 (local address 127.0.0.1:54338)
0926 05:51:34.368465 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:34.368515 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":81,"threads_started":1}
W20260926 05:51:34.368798 27327 catalog_manager.cc:4584] TS 681d141aa7c44470963e83a400f1362b (127.24.255.1:45807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 802a71a88a4547fbbb009bfdbfe43daf: Network error: Client connection negotiation failed: client connection to 127.24.255.1:45807: connect: Connection refused (error 111)
I20260926 05:51:34.373648 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 27695
I20260926 05:51:34.378623 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 27311
2026-09-26T05:51:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5961 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:34 dist-test-slave-7fr8 krb5kdc[27998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:34 dist-test-slave-7fr8 krb5kdc[27998](info): set up 2 sockets
Sep 26 05:51:34 dist-test-slave-7fr8 krb5kdc[27998](info): commencing operation
krb5kdc: starting...
W20260926 05:51:35.354398 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:35 dist-test-slave-7fr8 krb5kdc[27998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:51:35.473448 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:40073
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:34157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:40073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:35.559495 28011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:35.559725 28011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:35.559749 28011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:35.560951 28011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:35.560990 28011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:35.561004 28011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:35.561017 28011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:35.561028 28011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:35.562563 28011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:34157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:40073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:40073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:35.562783 28011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:35.562978 28011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:35.568626 28019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:35.568703 28016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:35.568665 28011 server_base.cc:1034] running on GCE node
W20260926 05:51:35.568627 28017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:35.569142 28011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:35.569444 28011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:35.570611 28011 hybrid_clock.cc:648] HybridClock initialized: now 1790401895570544 us; error 80 us; skew 500 ppm
Sep 26 05:51:35 dist-test-slave-7fr8 krb5kdc[27998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401895, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:35.585001 28011 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:51:35.585525 28011 webserver.cc:460] Webserver started at http://127.24.255.62:33061/ using document root <none> and password file <none>
I20260926 05:51:35.585763 28011 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:35.585811 28011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:35.585958 28011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:35.587134 28011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "fe30bd2d8c504858bbebbb1774627895"
format_stamp: "Formatted at 2026-09-26 05:51:35 on dist-test-slave-7fr8"
server_key: "a3647f531fea826c74b9faafa33ab35d"
server_key_iv: "0b78cd8e4d49a32955266c98c5721bfd"
server_key_version: "encryptionkey@0"
I20260926 05:51:35.587466 28011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "fe30bd2d8c504858bbebbb1774627895"
format_stamp: "Formatted at 2026-09-26 05:51:35 on dist-test-slave-7fr8"
server_key: "a3647f531fea826c74b9faafa33ab35d"
server_key_iv: "0b78cd8e4d49a32955266c98c5721bfd"
server_key_version: "encryptionkey@0"
I20260926 05:51:35.588917 28011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:35.589824 28026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:35.590278 28011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:35.590380 28011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "fe30bd2d8c504858bbebbb1774627895"
format_stamp: "Formatted at 2026-09-26 05:51:35 on dist-test-slave-7fr8"
server_key: "a3647f531fea826c74b9faafa33ab35d"
server_key_iv: "0b78cd8e4d49a32955266c98c5721bfd"
server_key_version: "encryptionkey@0"
I20260926 05:51:35.590456 28011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:35.607638 28011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:35.610150 28011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:35.610343 28011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:35.614799 28011 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:40073
I20260926 05:51:35.614979 28078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:40073 every 8 connection(s)
I20260926 05:51:35.615376 28011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:35.616325 28079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:35.618870 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 28011
I20260926 05:51:35.618954 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:35.619028 28079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895: Bootstrap starting.
I20260926 05:51:35.619110 25596 external_mini_cluster.cc:1428] Setting key 894e557935c0a8465e93d08589109977
I20260926 05:51:35.619704 28079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:35.620004 28079 log.cc:824] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:35.620769 28079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895: No bootstrap required, opened a new log
I20260926 05:51:35.622705 28079 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe30bd2d8c504858bbebbb1774627895" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 40073 } }
I20260926 05:51:35.622860 28079 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:35.622892 28079 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe30bd2d8c504858bbebbb1774627895, State: Initialized, Role: FOLLOWER
I20260926 05:51:35.623018 28079 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe30bd2d8c504858bbebbb1774627895" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 40073 } }
I20260926 05:51:35.623106 28079 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:35.623137 28079 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:35.623183 28079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:35.624039 28079 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe30bd2d8c504858bbebbb1774627895" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 40073 } }
I20260926 05:51:35.624167 28079 leader_election.cc:302] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe30bd2d8c504858bbebbb1774627895; no voters:
I20260926 05:51:35.624382 28079 leader_election.cc:288] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:35.624428 28084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:35.624632 28084 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [term 1 LEADER]: Becoming Leader. State: Replica: fe30bd2d8c504858bbebbb1774627895, State: Running, Role: LEADER
I20260926 05:51:35.624756 28079 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:35.624786 28084 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe30bd2d8c504858bbebbb1774627895" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 40073 } }
I20260926 05:51:35.625269 28084 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe30bd2d8c504858bbebbb1774627895. Latest consensus state: current_term: 1 leader_uuid: "fe30bd2d8c504858bbebbb1774627895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe30bd2d8c504858bbebbb1774627895" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 40073 } } }
I20260926 05:51:35.625387 28085 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe30bd2d8c504858bbebbb1774627895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe30bd2d8c504858bbebbb1774627895" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 40073 } } }
I20260926 05:51:35.625524 28085 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:35.625510 28084 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:35.625782 28093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 05:51:35 dist-test-slave-7fr8 krb5kdc[27998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:35.626557 28093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:35.628325 28093 catalog_manager.cc:1347] Generated new cluster ID: f1450fcc9f4a420a87b2eeeb7fff7561
I20260926 05:51:35.628378 28093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:35.630151 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:35.619678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.24.255.62:40073)
0926 05:51:35.619923 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:35.619931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:35.619957 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:51:35.620543 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:35.620551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:35.620605 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:35.620736 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:35.621356 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:35.622307 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:35.623063 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:35.623672 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:35.623810 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:35.626610 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:35.626614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:35.626615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:35.626628 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:35.628478 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:35.628909 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:35.628912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:35.628912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:35.629018 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:35.629213 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:35.629214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:35.629214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:35.629506 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:35.629655 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:35.629770 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 05:51:35.629909 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
I20260926 05:51:35.643024 28093 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:35.643839 28093 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:35.659668 28093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe30bd2d8c504858bbebbb1774627895: Generated new TSK 0
I20260926 05:51:35.659907 28093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:51:35.686338 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:40073
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:34157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:35.776167 28105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:35.776412 28105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:35.776439 28105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:35.777611 28105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:35.777654 28105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:35.777683 28105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:35.779250 28105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:34157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:40073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:35.779488 28105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:35.779686 28105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:35.784660 28110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:35.784695 28113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:35.784704 28111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:35.784854 28105 server_base.cc:1034] running on GCE node
I20260926 05:51:35.785058 28105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:35.785298 28105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:35.786489 28105 hybrid_clock.cc:648] HybridClock initialized: now 1790401895786473 us; error 30 us; skew 500 ppm
Sep 26 05:51:35 dist-test-slave-7fr8 krb5kdc[27998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401895, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:35.789090 28105 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:51:35.789539 28105 webserver.cc:460] Webserver started at http://127.24.255.1:46397/ using document root <none> and password file <none>
I20260926 05:51:35.789743 28105 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:35.789793 28105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:35.789922 28105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:35.790992 28105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "4724e0bf22af496483b0c11909ea0b9d"
format_stamp: "Formatted at 2026-09-26 05:51:35 on dist-test-slave-7fr8"
server_key: "02c178b5bcc7f6a807b1711a2c308296"
server_key_iv: "5640aa99e864cf9b37e45e84ad5e63d3"
server_key_version: "encryptionkey@0"
I20260926 05:51:35.791326 28105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "4724e0bf22af496483b0c11909ea0b9d"
format_stamp: "Formatted at 2026-09-26 05:51:35 on dist-test-slave-7fr8"
server_key: "02c178b5bcc7f6a807b1711a2c308296"
server_key_iv: "5640aa99e864cf9b37e45e84ad5e63d3"
server_key_version: "encryptionkey@0"
I20260926 05:51:35.792623 28105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:35.793401 28120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:35.793547 28105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:51:35.793630 28105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "4724e0bf22af496483b0c11909ea0b9d"
format_stamp: "Formatted at 2026-09-26 05:51:35 on dist-test-slave-7fr8"
server_key: "02c178b5bcc7f6a807b1711a2c308296"
server_key_iv: "5640aa99e864cf9b37e45e84ad5e63d3"
server_key_version: "encryptionkey@0"
I20260926 05:51:35.793699 28105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:35.813076 28105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:35.815549 28105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:35.815740 28105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:35.816057 28105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:35.816484 28105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:35.816522 28105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:35.816571 28105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:35.816596 28105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:35.822528 28105 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:38781
I20260926 05:51:35.822583 28233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:38781 every 8 connection(s)
I20260926 05:51:35.822908 28105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:51:35 dist-test-slave-7fr8 krb5kdc[27998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401895, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:35.831940 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 28105
I20260926 05:51:35.832031 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:51:35.832192 25596 external_mini_cluster.cc:1428] Setting key 28eb529f96eddc822d9b5b30061aa8bc
I20260926 05:51:35.832396 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:35.823847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:44723 (local address 127.24.255.62:40073)
0926 05:51:35.823944 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:35.823946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:35.824112 (+ 166us) server_negotiation.cc:407] Connection header received
0926 05:51:35.824682 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:35.824683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:35.824696 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:35.824725 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:35.825789 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:35.826323 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:35.827298 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:35.827503 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:35.827558 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:35.829814 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:35.829816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:35.829816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:35.829823 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:35.831007 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:35.831382 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:35.831383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:35.831383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:35.831438 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:35.831679 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:35.831680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:35.831680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:35.831829 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:35.831908 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:35.832218 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 05:51:35.832308 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 05:51:35.832687 28236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:35.823772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:40073 (local address 127.24.255.1:44723)
0926 05:51:35.823984 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:35.824031 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:35.824538 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:35.824881 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:35.824885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:35.825465 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:35.825685 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:35.825690 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:35.826412 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:35.826413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:35.827215 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:35.827218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:35.827574 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:35.827575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:35.827696 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:35.828276 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:35.828285 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:35.829721 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:35.831084 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:35.831090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:35.831093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:35.831314 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:35.831495 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:35.831495 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:35.831496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:35.831619 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:35.831919 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:35.831926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:35.832081 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 05:51:35.832191 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 05:51:35.832373 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":93,"threads_started":1}
I20260926 05:51:35.833184 28234 heartbeater.cc:347] Connected to a master server at 127.24.255.62:40073
I20260926 05:51:35.833273 28234 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:35.833455 28234 heartbeater.cc:511] Master 127.24.255.62:40073 requested a full tablet report, sending...
I20260926 05:51:35.833956 28043 ts_manager.cc:194] Registered new tserver with Master: 4724e0bf22af496483b0c11909ea0b9d (127.24.255.1:38781)
I20260926 05:51:35.834584 28043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:44723
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:51:35.856776 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:40073
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:34157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:35.940488 28239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:35.940716 28239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:35.940749 28239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:35.942214 28239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:35.942273 28239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:35.942317 28239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:35.943877 28239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:34157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:40073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:35.944132 28239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:35.944353 28239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:35.949359 28245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:35.949385 28247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:35.949366 28244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:35.949645 28239 server_base.cc:1034] running on GCE node
I20260926 05:51:35.949932 28239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:35.950193 28239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:35.951372 28239 hybrid_clock.cc:648] HybridClock initialized: now 1790401895951349 us; error 35 us; skew 500 ppm
Sep 26 05:51:35 dist-test-slave-7fr8 krb5kdc[27998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401895, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:35.953861 28239 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:51:35.954373 28239 webserver.cc:460] Webserver started at http://127.24.255.2:44421/ using document root <none> and password file <none>
I20260926 05:51:35.954563 28239 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:35.954602 28239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:35.954690 28239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:35.955724 28239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "f23492923e8d4fdea3d7a74b54b4be53"
format_stamp: "Formatted at 2026-09-26 05:51:35 on dist-test-slave-7fr8"
server_key: "3a3df0e5a5c0c17b294859c9a207e8de"
server_key_iv: "9b767ac22af06fb33df811b67ecc6330"
server_key_version: "encryptionkey@0"
I20260926 05:51:35.956033 28239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "f23492923e8d4fdea3d7a74b54b4be53"
format_stamp: "Formatted at 2026-09-26 05:51:35 on dist-test-slave-7fr8"
server_key: "3a3df0e5a5c0c17b294859c9a207e8de"
server_key_iv: "9b767ac22af06fb33df811b67ecc6330"
server_key_version: "encryptionkey@0"
I20260926 05:51:35.957309 28239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:51:35.958225 28254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:35.958393 28239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:51:35.958480 28239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "f23492923e8d4fdea3d7a74b54b4be53"
format_stamp: "Formatted at 2026-09-26 05:51:35 on dist-test-slave-7fr8"
server_key: "3a3df0e5a5c0c17b294859c9a207e8de"
server_key_iv: "9b767ac22af06fb33df811b67ecc6330"
server_key_version: "encryptionkey@0"
I20260926 05:51:35.958544 28239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:35.977268 28239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:35.979449 28239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:35.979612 28239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:35.979849 28239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:35.980288 28239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:35.980337 28239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:35.980423 28239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:35.980458 28239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:35.986035 28239 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:35079
I20260926 05:51:35.986119 28367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:35079 every 8 connection(s)
I20260926 05:51:35.986443 28239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:51:35.992202 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 28239
I20260926 05:51:35.992314 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:51:35.992503 25596 external_mini_cluster.cc:1428] Setting key 1017dacf8feaeb51036273e3882dc2f4
Sep 26 05:51:35 dist-test-slave-7fr8 krb5kdc[27998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401895, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:35.996423 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:35.987390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:59703 (local address 127.24.255.62:40073)
0926 05:51:35.987447 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:35.987449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:35.987529 (+ 80us) server_negotiation.cc:407] Connection header received
0926 05:51:35.988071 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:35.988072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:35.988087 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:35.988140 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:35.989131 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:35.989672 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:35.990910 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:35.991112 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:35.991181 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:35.993613 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:35.993615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:35.993616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:35.993623 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:35.995004 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:35.995509 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:35.995509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:35.995510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:35.995558 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:35.995799 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:35.995800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:35.995801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:35.995915 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:35.995972 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:35.996247 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 05:51:35.996330 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:51:35.996658 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:35.987154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:40073 (local address 127.24.255.2:59703)
0926 05:51:35.987392 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:35.987436 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:35.987945 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:35.988205 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:35.988208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:35.988813 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:35.989026 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:35.989031 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:35.989774 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:35.989774 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:35.990806 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:35.990809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:35.991219 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:35.991220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:35.991339 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:35.991885 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:35.991894 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:35.993480 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:35.995083 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:35.995091 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:35.995094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:35.995415 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:35.995628 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:35.995629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:35.995630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:35.995742 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:35.995974 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:35.995980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:35.996125 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 05:51:35.996217 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 05:51:35.996390 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260926 05:51:35.997125 28368 heartbeater.cc:347] Connected to a master server at 127.24.255.62:40073
I20260926 05:51:35.997223 28368 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:35.997398 28368 heartbeater.cc:511] Master 127.24.255.62:40073 requested a full tablet report, sending...
I20260926 05:51:35.997838 28043 ts_manager.cc:194] Registered new tserver with Master: f23492923e8d4fdea3d7a74b54b4be53 (127.24.255.2:35079)
I20260926 05:51:35.998267 28043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:59703
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:51:36.017211 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:40073
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:34157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:36.098070 28373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:36.098304 28373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:36.098328 28373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:36.099568 28373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:36.099619 28373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:36.099646 28373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:36.101119 28373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:34157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:40073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:36.101347 28373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:36.101567 28373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:36.106638 28379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:36.106654 28373 server_base.cc:1034] running on GCE node
W20260926 05:51:36.106724 28378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:36.106638 28381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:36.107043 28373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:36.107288 28373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:36.108462 28373 hybrid_clock.cc:648] HybridClock initialized: now 1790401896108451 us; error 33 us; skew 500 ppm
Sep 26 05:51:36 dist-test-slave-7fr8 krb5kdc[27998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401896, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:36.111004 28373 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:51:36.111502 28373 webserver.cc:460] Webserver started at http://127.24.255.3:39121/ using document root <none> and password file <none>
I20260926 05:51:36.111711 28373 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:36.111755 28373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:36.111869 28373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:36.112916 28373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "c2ed7883fd4e4b05b5834e10acff1490"
format_stamp: "Formatted at 2026-09-26 05:51:36 on dist-test-slave-7fr8"
server_key: "4439399f765a3e28a8b646c5a3baa4c0"
server_key_iv: "0f29e34ab859437533d0a63cc45ad1a6"
server_key_version: "encryptionkey@0"
I20260926 05:51:36.113247 28373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "c2ed7883fd4e4b05b5834e10acff1490"
format_stamp: "Formatted at 2026-09-26 05:51:36 on dist-test-slave-7fr8"
server_key: "4439399f765a3e28a8b646c5a3baa4c0"
server_key_iv: "0f29e34ab859437533d0a63cc45ad1a6"
server_key_version: "encryptionkey@0"
I20260926 05:51:36.114593 28373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:51:36.115338 28388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:36.115528 28373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:36.115608 28373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "c2ed7883fd4e4b05b5834e10acff1490"
format_stamp: "Formatted at 2026-09-26 05:51:36 on dist-test-slave-7fr8"
server_key: "4439399f765a3e28a8b646c5a3baa4c0"
server_key_iv: "0f29e34ab859437533d0a63cc45ad1a6"
server_key_version: "encryptionkey@0"
I20260926 05:51:36.115679 28373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:36.131140 28373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:36.133533 28373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:36.133671 28373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:36.133909 28373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:36.134260 28373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:36.134296 28373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:36.134325 28373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:36.134419 28373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:36.140722 28373 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:41189
I20260926 05:51:36.140765 28501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:41189 every 8 connection(s)
I20260926 05:51:36.141094 28373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:51:36.142557 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 28373
I20260926 05:51:36.142660 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:51:36.142851 25596 external_mini_cluster.cc:1428] Setting key 6e1313b55c701402829c6cef89908eea
Sep 26 05:51:36 dist-test-slave-7fr8 krb5kdc[27998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401896, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:36.150516 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:36.141819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:48289 (local address 127.24.255.62:40073)
0926 05:51:36.141881 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:36.141883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:36.142065 (+ 182us) server_negotiation.cc:407] Connection header received
0926 05:51:36.142605 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:36.142606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:36.142620 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:36.142672 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:36.143759 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:36.144273 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:36.145330 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:36.145556 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:36.145608 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:36.147820 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:36.147822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:36.147822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:36.147829 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:36.149059 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:36.149543 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:36.149544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:36.149544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:36.149592 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:36.149815 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:36.149816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:36.149816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:36.149919 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:36.149962 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:36.150335 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 05:51:36.150442 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:51:36.150604 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:36.141808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:40073 (local address 127.24.255.3:48289)
0926 05:51:36.141950 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:36.141986 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:36.142499 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:36.142760 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:36.142763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:36.143437 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:36.143681 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:36.143686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:36.144411 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:36.144413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:36.145255 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:36.145257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:36.145619 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:36.145620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:36.145737 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:36.146291 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:36.146300 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:36.147719 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:36.149146 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:36.149152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:36.149154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:36.149444 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:36.149654 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:36.149654 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:36.149655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:36.149763 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:36.149970 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:36.149975 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:36.150133 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 05:51:36.150231 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 05:51:36.150422 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260926 05:51:36.151173 28502 heartbeater.cc:347] Connected to a master server at 127.24.255.62:40073
I20260926 05:51:36.151337 28502 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:36.151539 28502 heartbeater.cc:511] Master 127.24.255.62:40073 requested a full tablet report, sending...
I20260926 05:51:36.152060 28043 ts_manager.cc:194] Registered new tserver with Master: c2ed7883fd4e4b05b5834e10acff1490 (127.24.255.3:41189)
I20260926 05:51:36.152468 28043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:48289
I20260926 05:51:36.154851 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 05:51:36.168320 28082 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:36.167119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.24.255.62:40073)
0926 05:51:36.167181 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:36.167184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:36.167535 (+ 351us) server_negotiation.cc:407] Connection header received
0926 05:51:36.167615 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:36.167616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:36.167628 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:36.167673 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:36.168234 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36646: BlockingRecv error: recv got EOF from 127.0.0.1:36646 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:51:36.169198 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 28105
I20260926 05:51:36.173424 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 28239
I20260926 05:51:36.177358 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 28373
I20260926 05:51:36.181185 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 28011
2026-09-26T05:51:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1801 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:36 dist-test-slave-7fr8 krb5kdc[28514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:36 dist-test-slave-7fr8 krb5kdc[28514](info): set up 2 sockets
Sep 26 05:51:36 dist-test-slave-7fr8 krb5kdc[28514](info): commencing operation
krb5kdc: starting...
W20260926 05:51:37.154038 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:37 dist-test-slave-7fr8 krb5kdc[28514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:51:37.277663 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:46601
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:46601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:37.367506 28527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:37.367736 28527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:37.367767 28527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:37.369004 28527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:37.369053 28527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:37.369072 28527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:37.369088 28527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:37.369104 28527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:37.370709 28527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:46601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:46601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:37.370970 28527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:37.371177 28527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:37.376183 28532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:37.376169 28533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:37.376170 28535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:37.376379 28527 server_base.cc:1034] running on GCE node
I20260926 05:51:37.376631 28527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:37.376930 28527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:37.378183 28527 hybrid_clock.cc:648] HybridClock initialized: now 1790401897378159 us; error 36 us; skew 500 ppm
Sep 26 05:51:37 dist-test-slave-7fr8 krb5kdc[28514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401897, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:37.381090 28527 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:51:37.381621 28527 webserver.cc:460] Webserver started at http://127.24.255.62:42451/ using document root <none> and password file <none>
I20260926 05:51:37.381831 28527 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:37.381880 28527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:37.382009 28527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:37.383148 28527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "a63b288965a44dd8bfa7e093100dbb1e"
format_stamp: "Formatted at 2026-09-26 05:51:37 on dist-test-slave-7fr8"
server_key: "79a90ac616a4b19d725ef00bf58abb1e"
server_key_iv: "382d0c45d5eccb7de6850756f65fa12a"
server_key_version: "encryptionkey@0"
I20260926 05:51:37.383498 28527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "a63b288965a44dd8bfa7e093100dbb1e"
format_stamp: "Formatted at 2026-09-26 05:51:37 on dist-test-slave-7fr8"
server_key: "79a90ac616a4b19d725ef00bf58abb1e"
server_key_iv: "382d0c45d5eccb7de6850756f65fa12a"
server_key_version: "encryptionkey@0"
I20260926 05:51:37.384981 28527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:37.385850 28542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:37.386051 28527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:37.386224 28527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "a63b288965a44dd8bfa7e093100dbb1e"
format_stamp: "Formatted at 2026-09-26 05:51:37 on dist-test-slave-7fr8"
server_key: "79a90ac616a4b19d725ef00bf58abb1e"
server_key_iv: "382d0c45d5eccb7de6850756f65fa12a"
server_key_version: "encryptionkey@0"
I20260926 05:51:37.386302 28527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:37.409579 28527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:37.411791 28527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:37.411937 28527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:37.415810 28527 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:46601
I20260926 05:51:37.415898 28594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:46601 every 8 connection(s)
I20260926 05:51:37.416322 28527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:37.416990 28595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:37.419422 28595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e: Bootstrap starting.
I20260926 05:51:37.419991 28595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:37.420229 28595 log.cc:824] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:37.421010 28595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e: No bootstrap required, opened a new log
I20260926 05:51:37.422811 28595 raft_consensus.cc:348] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63b288965a44dd8bfa7e093100dbb1e" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46601 } }
I20260926 05:51:37.422981 28595 raft_consensus.cc:374] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:37.423004 28595 raft_consensus.cc:729] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a63b288965a44dd8bfa7e093100dbb1e, State: Initialized, Role: FOLLOWER
I20260926 05:51:37.423121 28595 consensus_queue.cc:260] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63b288965a44dd8bfa7e093100dbb1e" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46601 } }
I20260926 05:51:37.423213 28595 raft_consensus.cc:388] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:37.423247 28595 raft_consensus.cc:482] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:37.423295 28595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:37.423456 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 28527
I20260926 05:51:37.423539 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:37.423698 25596 external_mini_cluster.cc:1428] Setting key 538320ec3c8e9bb75874da21dfa09134
I20260926 05:51:37.424675 28595 raft_consensus.cc:504] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63b288965a44dd8bfa7e093100dbb1e" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46601 } }
I20260926 05:51:37.424806 28595 leader_election.cc:302] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a63b288965a44dd8bfa7e093100dbb1e; no voters:
I20260926 05:51:37.425062 28595 leader_election.cc:288] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:37.425132 28600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:37.425303 28600 raft_consensus.cc:686] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [term 1 LEADER]: Becoming Leader. State: Replica: a63b288965a44dd8bfa7e093100dbb1e, State: Running, Role: LEADER
I20260926 05:51:37.425341 28595 sys_catalog.cc:564] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:37.426117 28600 consensus_queue.cc:237] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63b288965a44dd8bfa7e093100dbb1e" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46601 } }
I20260926 05:51:37.426716 28601 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a63b288965a44dd8bfa7e093100dbb1e. Latest consensus state: current_term: 1 leader_uuid: "a63b288965a44dd8bfa7e093100dbb1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63b288965a44dd8bfa7e093100dbb1e" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46601 } } }
I20260926 05:51:37.427177 28601 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [sys.catalog]: This master's current role is: LEADER
W20260926 05:51:37.427271 28614 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:51:37.427306 28614 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:37.426856 28602 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a63b288965a44dd8bfa7e093100dbb1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63b288965a44dd8bfa7e093100dbb1e" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46601 } } }
I20260926 05:51:37.427518 28602 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:37.428748 28616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:37.428941 28616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:37.430832 28616 catalog_manager.cc:1347] Generated new cluster ID: 69475ab847fd4991af088e5d8293d702
I20260926 05:51:37.430881 28616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:51:37 dist-test-slave-7fr8 krb5kdc[28514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:37.435901 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:37.424238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.24.255.62:46601)
0926 05:51:37.424429 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:37.424438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:37.424465 (+ 27us) server_negotiation.cc:407] Connection header received
0926 05:51:37.424566 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:37.424572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:37.424617 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:37.424728 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:37.425608 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:37.427502 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:37.428647 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:37.429179 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:37.429288 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:37.432160 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:37.432165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:37.432166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:37.432180 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:37.434208 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:37.434602 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:37.434606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:37.434606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:37.434741 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:37.434943 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:37.434944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:37.434944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:37.435289 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:37.435458 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:37.435530 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 05:51:37.435684 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260926 05:51:37.437238 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:37.441460 28616 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:37.442325 28616 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:37.448210 28616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a63b288965a44dd8bfa7e093100dbb1e: Generated new TSK 0
I20260926 05:51:37.448447 28616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:37.452816 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:37.449446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.24.255.62:46601)
0926 05:51:37.449505 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:37.449507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:37.449608 (+ 101us) server_negotiation.cc:407] Connection header received
0926 05:51:37.449677 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:37.449678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:37.449689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:37.449730 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:37.450132 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:37.450670 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:37.451409 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:37.451643 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:37.451700 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:37.452364 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:37.452366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:37.452366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:37.452563 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:37.452582 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:37.452587 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:37.452633 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:37.452698 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 05:51:37.452734 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:51:37.455178 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 28527
2026-09-26T05:51:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1271 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:37 dist-test-slave-7fr8 krb5kdc[28628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:37 dist-test-slave-7fr8 krb5kdc[28628](info): set up 2 sockets
Sep 26 05:51:37 dist-test-slave-7fr8 krb5kdc[28628](info): commencing operation
krb5kdc: starting...
W20260926 05:51:38.422487 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:38 dist-test-slave-7fr8 krb5kdc[28628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:51:38.555675 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:33869
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:33869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:38.640129 28641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:38.640374 28641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:38.640410 28641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:38.641645 28641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:38.641705 28641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:38.641727 28641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:38.641746 28641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:38.641774 28641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:38.643431 28641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:33869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:33869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:38.643716 28641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:38.644086 28641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:51:38.649897 28641 server_base.cc:1034] running on GCE node
W20260926 05:51:38.649910 28649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:38.649897 28647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:38.649928 28646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:38.650470 28641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:38.650768 28641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:38.651942 28641 hybrid_clock.cc:648] HybridClock initialized: now 1790401898651929 us; error 37 us; skew 500 ppm
Sep 26 05:51:38 dist-test-slave-7fr8 krb5kdc[28628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401898, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:38.654596 28641 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:51:38.655026 28641 webserver.cc:460] Webserver started at http://127.24.255.62:45001/ using document root <none> and password file <none>
I20260926 05:51:38.655237 28641 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:38.655292 28641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:38.655419 28641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:38.656469 28641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "4c957e5841cf4842976c8c226e27e5f9"
format_stamp: "Formatted at 2026-09-26 05:51:38 on dist-test-slave-7fr8"
server_key: "c1c49c47d272cb267a5d7f59b7e65237"
server_key_iv: "38fca4b3567a58409fdef359d6e042ed"
server_key_version: "encryptionkey@0"
I20260926 05:51:38.656817 28641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "4c957e5841cf4842976c8c226e27e5f9"
format_stamp: "Formatted at 2026-09-26 05:51:38 on dist-test-slave-7fr8"
server_key: "c1c49c47d272cb267a5d7f59b7e65237"
server_key_iv: "38fca4b3567a58409fdef359d6e042ed"
server_key_version: "encryptionkey@0"
I20260926 05:51:38.658164 28641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:38.659025 28656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:38.659179 28641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:38.659263 28641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "4c957e5841cf4842976c8c226e27e5f9"
format_stamp: "Formatted at 2026-09-26 05:51:38 on dist-test-slave-7fr8"
server_key: "c1c49c47d272cb267a5d7f59b7e65237"
server_key_iv: "38fca4b3567a58409fdef359d6e042ed"
server_key_version: "encryptionkey@0"
I20260926 05:51:38.659334 28641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:38.675063 28641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:38.677197 28641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:38.677340 28641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:38.681553 28641 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:33869
I20260926 05:51:38.681597 28708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:33869 every 8 connection(s)
I20260926 05:51:38.681933 28641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:38.682932 28709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:38.686512 28709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9: Bootstrap starting.
I20260926 05:51:38.687134 28709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:38.687405 28709 log.cc:824] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:38.688205 28709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9: No bootstrap required, opened a new log
I20260926 05:51:38.689980 28709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c957e5841cf4842976c8c226e27e5f9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33869 } }
I20260926 05:51:38.690167 28709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:38.690191 28709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c957e5841cf4842976c8c226e27e5f9, State: Initialized, Role: FOLLOWER
I20260926 05:51:38.690344 28709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c957e5841cf4842976c8c226e27e5f9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33869 } }
I20260926 05:51:38.690412 28709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:38.690433 28709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:38.690480 28709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:38.690850 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 28641
I20260926 05:51:38.690935 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:38.691095 25596 external_mini_cluster.cc:1428] Setting key ebeeb66df858e10c507755739dcc781d
I20260926 05:51:38.691700 28709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c957e5841cf4842976c8c226e27e5f9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33869 } }
I20260926 05:51:38.691819 28709 leader_election.cc:302] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c957e5841cf4842976c8c226e27e5f9; no voters:
I20260926 05:51:38.691973 28709 leader_election.cc:288] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:38.691999 28714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:38.692159 28714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [term 1 LEADER]: Becoming Leader. State: Replica: 4c957e5841cf4842976c8c226e27e5f9, State: Running, Role: LEADER
I20260926 05:51:38.692229 28709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:38.692291 28714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c957e5841cf4842976c8c226e27e5f9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33869 } }
I20260926 05:51:38.692699 28715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c957e5841cf4842976c8c226e27e5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c957e5841cf4842976c8c226e27e5f9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33869 } } }
I20260926 05:51:38.692780 28715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:38.692956 28716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c957e5841cf4842976c8c226e27e5f9. Latest consensus state: current_term: 1 leader_uuid: "4c957e5841cf4842976c8c226e27e5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c957e5841cf4842976c8c226e27e5f9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33869 } } }
I20260926 05:51:38.693029 28716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9 [sys.catalog]: This master's current role is: LEADER
W20260926 05:51:38.694193 28730 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:51:38.694240 28730 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:38.694281 28722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:38.694482 28722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:38.696069 28722 catalog_manager.cc:1347] Generated new cluster ID: c4330ca9d8064ad09976e95012e1f5a1
I20260926 05:51:38.696122 28722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:51:38 dist-test-slave-7fr8 krb5kdc[28628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:38.703548 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:38.691597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.24.255.62:33869)
0926 05:51:38.691808 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:38.691818 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:38.691867 (+ 49us) server_negotiation.cc:407] Connection header received
0926 05:51:38.692001 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:38.692009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:38.692075 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:38.693027 (+ 952us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:38.693127 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:38.694598 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:38.695537 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:38.696172 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:38.696280 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:38.699120 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:38.699125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:38.699125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:38.699140 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:38.701801 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:38.702260 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:38.702263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:38.702263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:38.702391 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:38.702584 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:38.702584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:38.702585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:38.702893 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:38.703056 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:38.703154 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 05:51:38.703300 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260926 05:51:38.711856 28722 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:38.712625 28722 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:38.719295 28722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c957e5841cf4842976c8c226e27e5f9: Generated new TSK 0
I20260926 05:51:38.719543 28722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:51:38.754349 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:33869
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:38.836905 28735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:38.837142 28735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:38.837167 28735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:38.838405 28735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:38.838452 28735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:38.838488 28735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:38.840075 28735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:38.840337 28735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:38.840574 28735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:38.845760 28740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:38.845773 28741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:38.845877 28735 server_base.cc:1034] running on GCE node
W20260926 05:51:38.845757 28743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:38.846256 28735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:38.846493 28735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:38.847630 28735 hybrid_clock.cc:648] HybridClock initialized: now 1790401898847594 us; error 55 us; skew 500 ppm
Sep 26 05:51:38 dist-test-slave-7fr8 krb5kdc[28628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401898, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:38.850258 28735 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:51:38.850734 28735 webserver.cc:460] Webserver started at http://127.24.255.1:41189/ using document root <none> and password file <none>
I20260926 05:51:38.850952 28735 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:38.850998 28735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:38.851130 28735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:38.852200 28735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "fab93a7a78d94f51b1fc6e57d7fc9a50"
format_stamp: "Formatted at 2026-09-26 05:51:38 on dist-test-slave-7fr8"
server_key: "709498abbdbe9e5ae956a2a1f88a6eba"
server_key_iv: "781ecc1868cc8a51e657151d4645dabb"
server_key_version: "encryptionkey@0"
I20260926 05:51:38.852550 28735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "fab93a7a78d94f51b1fc6e57d7fc9a50"
format_stamp: "Formatted at 2026-09-26 05:51:38 on dist-test-slave-7fr8"
server_key: "709498abbdbe9e5ae956a2a1f88a6eba"
server_key_iv: "781ecc1868cc8a51e657151d4645dabb"
server_key_version: "encryptionkey@0"
I20260926 05:51:38.853995 28735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:38.854830 28750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:38.854978 28735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:51:38.855058 28735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "fab93a7a78d94f51b1fc6e57d7fc9a50"
format_stamp: "Formatted at 2026-09-26 05:51:38 on dist-test-slave-7fr8"
server_key: "709498abbdbe9e5ae956a2a1f88a6eba"
server_key_iv: "781ecc1868cc8a51e657151d4645dabb"
server_key_version: "encryptionkey@0"
I20260926 05:51:38.855127 28735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:38.863445 28735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:38.865417 28735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:38.865535 28735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:38.865763 28735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:38.866123 28735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:38.866171 28735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:38.866194 28735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:38.866209 28735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:38.872232 28735 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:46467
I20260926 05:51:38.872280 28863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:46467 every 8 connection(s)
I20260926 05:51:38.872617 28735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:51:38 dist-test-slave-7fr8 krb5kdc[28628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401898, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:38.879715 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 28735
I20260926 05:51:38.879797 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:51:38.880002 25596 external_mini_cluster.cc:1428] Setting key 5abeb2819794b470c37c888bd2a04490
I20260926 05:51:38.881909 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:38.873341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:43757 (local address 127.24.255.62:33869)
0926 05:51:38.873395 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:38.873397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:38.873569 (+ 172us) server_negotiation.cc:407] Connection header received
0926 05:51:38.874046 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:38.874047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:38.874061 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:38.874184 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:38.875126 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:38.875648 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:38.876646 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:38.876843 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:38.876907 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:38.879123 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:38.879124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:38.879125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:38.879132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:38.880501 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:38.880936 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:38.880938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:38.880938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:38.881005 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:38.881248 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:38.881249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:38.881249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:38.881386 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:38.881447 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:38.881733 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 05:51:38.881819 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 05:51:38.882169 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:38.873266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:33869 (local address 127.24.255.1:43757)
0926 05:51:38.873428 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:38.873472 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:38.873952 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:38.874249 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:38.874253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:38.874801 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:38.875015 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:38.875019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:38.875742 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:38.875743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:38.876563 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:38.876566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:38.876963 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:38.876965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:38.877091 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:38.877567 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:38.877576 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:38.879014 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:38.880582 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:38.880592 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:38.880594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:38.880848 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:38.881075 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:38.881076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:38.881076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:38.881193 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:38.881467 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:38.881474 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:38.881607 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 05:51:38.881705 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 05:51:38.881876 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260926 05:51:38.882658 28864 heartbeater.cc:347] Connected to a master server at 127.24.255.62:33869
I20260926 05:51:38.882758 28864 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:38.882939 28864 heartbeater.cc:511] Master 127.24.255.62:33869 requested a full tablet report, sending...
I20260926 05:51:38.883491 28673 ts_manager.cc:194] Registered new tserver with Master: fab93a7a78d94f51b1fc6e57d7fc9a50 (127.24.255.1:46467)
I20260926 05:51:38.884332 28673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:43757
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:51:38.909253 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:33869
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:38.998672 28869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:38.998906 28869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:38.998943 28869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:39.000206 28869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:39.000247 28869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:39.000273 28869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:39.001823 28869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:39.002053 28869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:39.002293 28869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:39.007534 28874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:39.007540 28877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:39.007596 28869 server_base.cc:1034] running on GCE node
W20260926 05:51:39.007596 28875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:39.008033 28869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:39.008263 28869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:39.009431 28869 hybrid_clock.cc:648] HybridClock initialized: now 1790401899009393 us; error 88 us; skew 500 ppm
Sep 26 05:51:39 dist-test-slave-7fr8 krb5kdc[28628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401899, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:39.012387 28869 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:51:39.012895 28869 webserver.cc:460] Webserver started at http://127.24.255.2:42579/ using document root <none> and password file <none>
I20260926 05:51:39.013144 28869 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:39.013197 28869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:39.013305 28869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:39.014478 28869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "8d17e8f81c454945829e3d42b41a0dfa"
format_stamp: "Formatted at 2026-09-26 05:51:39 on dist-test-slave-7fr8"
server_key: "591bdf9463f3570c160f2a17e776a824"
server_key_iv: "a5d67b53c0ec6101fe02a8c46f9163fe"
server_key_version: "encryptionkey@0"
I20260926 05:51:39.014812 28869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "8d17e8f81c454945829e3d42b41a0dfa"
format_stamp: "Formatted at 2026-09-26 05:51:39 on dist-test-slave-7fr8"
server_key: "591bdf9463f3570c160f2a17e776a824"
server_key_iv: "a5d67b53c0ec6101fe02a8c46f9163fe"
server_key_version: "encryptionkey@0"
I20260926 05:51:39.016301 28869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:39.017443 28884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:39.017665 28869 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:39.017767 28869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "8d17e8f81c454945829e3d42b41a0dfa"
format_stamp: "Formatted at 2026-09-26 05:51:39 on dist-test-slave-7fr8"
server_key: "591bdf9463f3570c160f2a17e776a824"
server_key_iv: "a5d67b53c0ec6101fe02a8c46f9163fe"
server_key_version: "encryptionkey@0"
I20260926 05:51:39.017845 28869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:39.033234 28869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:39.035459 28869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:39.035620 28869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:39.035854 28869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:39.036190 28869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:39.036224 28869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:39.036254 28869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:39.036273 28869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:39.041751 28869 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:37827
I20260926 05:51:39.041802 28997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:37827 every 8 connection(s)
I20260926 05:51:39.042192 28869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:51:39.044735 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 28869
I20260926 05:51:39.044842 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:51:39.045032 25596 external_mini_cluster.cc:1428] Setting key 7331f5be49d97d263c25003dcd5c820e
Sep 26 05:51:39 dist-test-slave-7fr8 krb5kdc[28628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401899, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:39.052636 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:39.042916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:45721 (local address 127.24.255.62:33869)
0926 05:51:39.042996 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:39.042998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:39.043157 (+ 159us) server_negotiation.cc:407] Connection header received
0926 05:51:39.043659 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:39.043660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:39.043669 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:39.043716 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:39.044906 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:39.045638 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:39.046723 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:39.046989 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:39.047058 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:39.049568 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:39.049570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:39.049570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:39.049578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:39.050974 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:39.051476 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:39.051477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:39.051478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:39.051537 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:39.051844 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:39.051845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:39.051846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:39.051991 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:39.052056 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:39.052434 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 05:51:39.052565 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:51:39.052819 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:39.042871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:33869 (local address 127.24.255.2:45721)
0926 05:51:39.043043 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:39.043081 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:39.043579 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:39.043771 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:39.043774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:39.044478 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:39.044773 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:39.044780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:39.045719 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:39.045720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:39.046641 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:39.046644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:39.047093 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:39.047095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:39.047253 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:39.047825 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:39.047838 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:39.049455 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:39.051051 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:39.051062 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:39.051064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:39.051386 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:39.051603 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:39.051604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:39.051604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:39.051742 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:39.052065 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:39.052071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:39.052254 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 05:51:39.052396 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 05:51:39.052578 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260926 05:51:39.053442 28998 heartbeater.cc:347] Connected to a master server at 127.24.255.62:33869
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
I20260926 05:51:39.053552 28998 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:39.053736 28998 heartbeater.cc:511] Master 127.24.255.62:33869 requested a full tablet report, sending...
I20260926 05:51:39.054198 28673 ts_manager.cc:194] Registered new tserver with Master: 8d17e8f81c454945829e3d42b41a0dfa (127.24.255.2:37827)
I20260926 05:51:39.054577 28673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:45721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:51:39.074316 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:33869
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:39.160195 29003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:39.160418 29003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:39.160442 29003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:39.161646 29003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:39.161700 29003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:39.161726 29003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:39.163254 29003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:39.163511 29003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:39.163725 29003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:39.168970 29008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:39.168973 29011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:39.168972 29009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:39.169045 29003 server_base.cc:1034] running on GCE node
I20260926 05:51:39.169658 29003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:39.170238 29003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:39.171454 29003 hybrid_clock.cc:648] HybridClock initialized: now 1790401899171435 us; error 37 us; skew 500 ppm
Sep 26 05:51:39 dist-test-slave-7fr8 krb5kdc[28628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401899, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:39.174168 29003 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:51:39.174679 29003 webserver.cc:460] Webserver started at http://127.24.255.3:32867/ using document root <none> and password file <none>
I20260926 05:51:39.174892 29003 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:39.174942 29003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:39.175066 29003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:39.176193 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "b4d90a81d08a45ce85ca5d0fbb9a70eb"
format_stamp: "Formatted at 2026-09-26 05:51:39 on dist-test-slave-7fr8"
server_key: "318bc43ca0ca425c89fce753183e6898"
server_key_iv: "7cd79916cbcc0fc7fc4b52c9240a0fa1"
server_key_version: "encryptionkey@0"
I20260926 05:51:39.176568 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "b4d90a81d08a45ce85ca5d0fbb9a70eb"
format_stamp: "Formatted at 2026-09-26 05:51:39 on dist-test-slave-7fr8"
server_key: "318bc43ca0ca425c89fce753183e6898"
server_key_iv: "7cd79916cbcc0fc7fc4b52c9240a0fa1"
server_key_version: "encryptionkey@0"
I20260926 05:51:39.178392 29003 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 05:51:39.179379 29018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:39.179565 29003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:39.179661 29003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "b4d90a81d08a45ce85ca5d0fbb9a70eb"
format_stamp: "Formatted at 2026-09-26 05:51:39 on dist-test-slave-7fr8"
server_key: "318bc43ca0ca425c89fce753183e6898"
server_key_iv: "7cd79916cbcc0fc7fc4b52c9240a0fa1"
server_key_version: "encryptionkey@0"
I20260926 05:51:39.179741 29003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:39.194864 29003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:39.197093 29003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:39.197212 29003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:39.197471 29003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:39.197784 29003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:39.197820 29003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:39.197842 29003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:39.197854 29003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:39.203382 29003 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:45877
I20260926 05:51:39.203423 29131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:45877 every 8 connection(s)
I20260926 05:51:39.203802 29003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:51:39.209794 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 29003
I20260926 05:51:39.209887 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
Sep 26 05:51:39 dist-test-slave-7fr8 krb5kdc[28628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401899, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:39.210063 25596 external_mini_cluster.cc:1428] Setting key 1ba1ee168ae06876a3d6cd79321442b2
I20260926 05:51:39.213654 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:39.204450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:35847 (local address 127.24.255.62:33869)
0926 05:51:39.204527 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:39.204529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:39.204696 (+ 167us) server_negotiation.cc:407] Connection header received
0926 05:51:39.205170 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:39.205171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:39.205181 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:39.205230 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:39.206198 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:39.206715 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:39.207807 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:39.208038 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:39.208100 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:39.210648 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:39.210650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:39.210651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:39.210660 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:39.212056 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:39.212599 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:39.212600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:39.212600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:39.212672 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:39.212975 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:39.212976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:39.212976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:39.213107 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:39.213169 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:39.213510 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 05:51:39.213586 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:51:39.213740 29134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:39.204423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:33869 (local address 127.24.255.3:35847)
0926 05:51:39.204570 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:39.204606 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:39.205089 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:39.205284 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:39.205288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:39.205861 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:39.206063 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:39.206068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:39.206869 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:39.206869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:39.207698 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:39.207700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:39.208140 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:39.208141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:39.208266 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:39.208804 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:39.208813 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:39.210513 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:39.212185 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:39.212200 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:39.212202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:39.212509 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:39.212742 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:39.212743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:39.212743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:39.212892 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:39.213162 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:39.213167 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:39.213333 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 05:51:39.213423 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 05:51:39.213558 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260926 05:51:39.214206 29132 heartbeater.cc:347] Connected to a master server at 127.24.255.62:33869
I20260926 05:51:39.214313 29132 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:39.214509 29132 heartbeater.cc:511] Master 127.24.255.62:33869 requested a full tablet report, sending...
I20260926 05:51:39.214958 28673 ts_manager.cc:194] Registered new tserver with Master: b4d90a81d08a45ce85ca5d0fbb9a70eb (127.24.255.3:45877)
I20260926 05:51:39.215481 28673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:35847
I20260926 05:51:39.222911 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 05:51:39 dist-test-slave-7fr8 krb5kdc[28628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401899, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 05:51:39 dist-test-slave-7fr8 krb5kdc[28628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401899, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:39.246186 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:39.238890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.24.255.62:33869)
0926 05:51:39.238996 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:39.238998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:39.239009 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:51:39.239088 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:39.239089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:39.239099 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:39.239146 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:39.239869 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:39.240363 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:39.241094 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:39.241290 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:39.241347 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:39.243612 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:39.243614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:39.243614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:39.243622 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:39.245042 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:39.245390 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:39.245391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:39.245392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:39.245454 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:39.245614 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:39.245614 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:39.245615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:39.245774 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:39.245865 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:39.245980 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 05:51:39.246064 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 05:51:39.246409 28673 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34086
I20260926 05:51:39.247174 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 28735
I20260926 05:51:39.252048 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 28869
I20260926 05:51:39.256328 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 29003
I20260926 05:51:39.260702 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 28641
2026-09-26T05:51:39Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1807 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:39 dist-test-slave-7fr8 krb5kdc[29144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:39 dist-test-slave-7fr8 krb5kdc[29144](info): set up 2 sockets
Sep 26 05:51:39 dist-test-slave-7fr8 krb5kdc[29144](info): commencing operation
krb5kdc: starting...
W20260926 05:51:40.230170 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:40 dist-test-slave-7fr8 krb5kdc[29144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:51:40.361037 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:38801
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:37071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:38801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:40.442485 29157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:40.442709 29157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:40.442734 29157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:40.444024 29157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:40.444084 29157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:40.444098 29157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:40.444110 29157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:40.444121 29157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:40.445844 29157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:38801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:38801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:40.446265 29157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:40.446530 29157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:40.451772 29163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:40.451802 29165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:40.451840 29157 server_base.cc:1034] running on GCE node
W20260926 05:51:40.451808 29162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:40.452214 29157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:40.452533 29157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:40.453704 29157 hybrid_clock.cc:648] HybridClock initialized: now 1790401900453676 us; error 41 us; skew 500 ppm
Sep 26 05:51:40 dist-test-slave-7fr8 krb5kdc[29144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401900, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:40.456866 29157 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:51:40.457322 29157 webserver.cc:460] Webserver started at http://127.24.255.62:37757/ using document root <none> and password file <none>
I20260926 05:51:40.457530 29157 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:40.457573 29157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:40.457664 29157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:40.458778 29157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "03e8e04d53114e27a156c1457976ab09"
format_stamp: "Formatted at 2026-09-26 05:51:40 on dist-test-slave-7fr8"
server_key: "ebba93680b58dfb5b90a79e93f46bbf6"
server_key_iv: "ae8c9585e1937ddb6789b4e7adb6ab88"
server_key_version: "encryptionkey@0"
I20260926 05:51:40.459095 29157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "03e8e04d53114e27a156c1457976ab09"
format_stamp: "Formatted at 2026-09-26 05:51:40 on dist-test-slave-7fr8"
server_key: "ebba93680b58dfb5b90a79e93f46bbf6"
server_key_iv: "ae8c9585e1937ddb6789b4e7adb6ab88"
server_key_version: "encryptionkey@0"
I20260926 05:51:40.460453 29157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:40.461347 29172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:40.461553 29157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:40.461633 29157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "03e8e04d53114e27a156c1457976ab09"
format_stamp: "Formatted at 2026-09-26 05:51:40 on dist-test-slave-7fr8"
server_key: "ebba93680b58dfb5b90a79e93f46bbf6"
server_key_iv: "ae8c9585e1937ddb6789b4e7adb6ab88"
server_key_version: "encryptionkey@0"
I20260926 05:51:40.461710 29157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:40.472869 29157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:40.475368 29157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:40.475498 29157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:40.479353 29157 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:38801
I20260926 05:51:40.479722 29157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:40.479909 29224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:38801 every 8 connection(s)
I20260926 05:51:40.480479 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:40.482980 29225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09: Bootstrap starting.
I20260926 05:51:40.483608 29225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:40.483888 29225 log.cc:824] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:40.484576 29225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09: No bootstrap required, opened a new log
I20260926 05:51:40.486235 29225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e8e04d53114e27a156c1457976ab09" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38801 } }
I20260926 05:51:40.486399 29225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:40.486433 29225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e8e04d53114e27a156c1457976ab09, State: Initialized, Role: FOLLOWER
I20260926 05:51:40.486521 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 29157
I20260926 05:51:40.486611 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:40.486579 29225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e8e04d53114e27a156c1457976ab09" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38801 } }
I20260926 05:51:40.486655 29225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:40.486694 29225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:40.486742 29225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:40.486795 25596 external_mini_cluster.cc:1428] Setting key c190b9422172f59f932053c3156c91dc
I20260926 05:51:40.487900 29225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e8e04d53114e27a156c1457976ab09" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38801 } }
I20260926 05:51:40.488036 29225 leader_election.cc:302] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03e8e04d53114e27a156c1457976ab09; no voters:
I20260926 05:51:40.488201 29225 leader_election.cc:288] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:40.488238 29230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:40.488348 29230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [term 1 LEADER]: Becoming Leader. State: Replica: 03e8e04d53114e27a156c1457976ab09, State: Running, Role: LEADER
I20260926 05:51:40.488432 29230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e8e04d53114e27a156c1457976ab09" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38801 } }
I20260926 05:51:40.488473 29225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:40.489653 29232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03e8e04d53114e27a156c1457976ab09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e8e04d53114e27a156c1457976ab09" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38801 } } }
I20260926 05:51:40.489761 29232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [sys.catalog]: This master's current role is: LEADER
W20260926 05:51:40.490307 29245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:51:40.490353 29245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:40.490485 29231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03e8e04d53114e27a156c1457976ab09. Latest consensus state: current_term: 1 leader_uuid: "03e8e04d53114e27a156c1457976ab09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e8e04d53114e27a156c1457976ab09" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38801 } } }
I20260926 05:51:40.490540 29231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:40.490634 29246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:40.490774 29246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 05:51:40 dist-test-slave-7fr8 krb5kdc[29144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:40.494959 29246 catalog_manager.cc:1347] Generated new cluster ID: 7cd81f6a1cd548c2bbead9dc8a65a8e0
I20260926 05:51:40.495005 29246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:40.499163 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:40.487380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.24.255.62:38801)
0926 05:51:40.487606 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:40.487616 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:40.487657 (+ 41us) server_negotiation.cc:407] Connection header received
0926 05:51:40.487783 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:40.487790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:40.487862 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:40.488013 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:40.488784 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:40.490263 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:40.491231 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:40.491956 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:40.492080 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:40.494673 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:40.494677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:40.494677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:40.494689 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:40.496655 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:40.497082 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:40.497086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:40.497086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:40.497291 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:40.497496 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:40.497497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:40.497497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:40.497837 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:40.497984 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:40.498517 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 05:51:40.498665 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":106,"threads_started":1}
I20260926 05:51:40.511469 29246 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:40.512035 29246 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:40.518996 29246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03e8e04d53114e27a156c1457976ab09: Generated new TSK 0
I20260926 05:51:40.519205 29246 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:51:40.557811 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38801
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:37071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:40.645689 29251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:40.645922 29251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:40.645948 29251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:40.647286 29251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:40.647338 29251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:40.647369 29251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:40.648968 29251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:40.649206 29251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:40.649396 29251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:40.654960 29259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:40.654966 29256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:40.654981 29257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:40.654955 29251 server_base.cc:1034] running on GCE node
I20260926 05:51:40.655385 29251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:40.655633 29251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:40.656802 29251 hybrid_clock.cc:648] HybridClock initialized: now 1790401900656787 us; error 28 us; skew 500 ppm
Sep 26 05:51:40 dist-test-slave-7fr8 krb5kdc[29144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401900, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:40.664511 29251 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:51:40.665105 29251 webserver.cc:460] Webserver started at http://127.24.255.1:45517/ using document root <none> and password file <none>
I20260926 05:51:40.665333 29251 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:40.665382 29251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:40.665524 29251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:40.666726 29251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "58f8864b6d2949da82758b7484c556aa"
format_stamp: "Formatted at 2026-09-26 05:51:40 on dist-test-slave-7fr8"
server_key: "5662d4161922d1f5c2e95293a5e19ba9"
server_key_iv: "7beb99c3513246f077105e283bcebeca"
server_key_version: "encryptionkey@0"
I20260926 05:51:40.667068 29251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "58f8864b6d2949da82758b7484c556aa"
format_stamp: "Formatted at 2026-09-26 05:51:40 on dist-test-slave-7fr8"
server_key: "5662d4161922d1f5c2e95293a5e19ba9"
server_key_iv: "7beb99c3513246f077105e283bcebeca"
server_key_version: "encryptionkey@0"
I20260926 05:51:40.668437 29251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:40.669217 29266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:40.669482 29251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:40.669570 29251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "58f8864b6d2949da82758b7484c556aa"
format_stamp: "Formatted at 2026-09-26 05:51:40 on dist-test-slave-7fr8"
server_key: "5662d4161922d1f5c2e95293a5e19ba9"
server_key_iv: "7beb99c3513246f077105e283bcebeca"
server_key_version: "encryptionkey@0"
I20260926 05:51:40.669633 29251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:40.694620 29251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:40.696583 29251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:40.696722 29251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:40.696939 29251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:40.697237 29251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:40.697270 29251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:40.697299 29251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:40.697324 29251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:40.702818 29251 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:39567
I20260926 05:51:40.702941 29379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:39567 every 8 connection(s)
I20260926 05:51:40.703186 29251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:51:40.703724 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 29251
I20260926 05:51:40.703827 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:51:40.704052 25596 external_mini_cluster.cc:1428] Setting key 7c48fe3c3308fbdfe8c378b98fcbb183
Sep 26 05:51:40 dist-test-slave-7fr8 krb5kdc[29144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401900, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
I20260926 05:51:40.713308 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:40.704250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:52891 (local address 127.24.255.62:38801)
0926 05:51:40.704296 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:40.704298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:40.704308 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:40.705375 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:40.705377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:40.705392 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:40.705455 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:40.706152 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:40.706668 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:40.707703 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:40.707904 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:40.707962 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:40.710505 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:40.710507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:40.710508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:40.710516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:40.711792 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:40.712346 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:40.712347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:40.712348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:40.712405 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:40.712629 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:40.712629 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:40.712630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:40.712739 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:40.712785 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:40.713128 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 05:51:40.713237 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:51:40.713434 29382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:40.703952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38801 (local address 127.24.255.1:52891)
0926 05:51:40.704181 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:40.704232 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:40.705267 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:40.705538 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:40.705542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:40.705798 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:40.706044 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:40.706049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:40.706756 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:40.706757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:40.707604 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:40.707608 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:40.707981 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:40.707982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:40.708117 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:40.708677 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:40.708693 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:40.710314 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:40.711933 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:40.711943 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:40.711946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:40.712268 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:40.712471 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:40.712472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:40.712472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:40.712581 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:40.712807 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:40.712813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:40.712980 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 05:51:40.713087 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 05:51:40.713242 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
I20260926 05:51:40.713932 29380 heartbeater.cc:347] Connected to a master server at 127.24.255.62:38801
I20260926 05:51:40.714125 29380 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:40.714339 29380 heartbeater.cc:511] Master 127.24.255.62:38801 requested a full tablet report, sending...
I20260926 05:51:40.714874 29189 ts_manager.cc:194] Registered new tserver with Master: 58f8864b6d2949da82758b7484c556aa (127.24.255.1:39567)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
I20260926 05:51:40.715515 29189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:52891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:51:40.729597 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38801
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:37071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:40.812444 29385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:40.812673 29385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:40.812700 29385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:40.814024 29385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:40.814090 29385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:40.814169 29385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:40.816310 29385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:40.816610 29385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:40.816869 29385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:40.822073 29390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:40.822263 29393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:40.822072 29391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:40.822482 29385 server_base.cc:1034] running on GCE node
I20260926 05:51:40.822690 29385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:40.822911 29385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:40.824062 29385 hybrid_clock.cc:648] HybridClock initialized: now 1790401900824042 us; error 38 us; skew 500 ppm
Sep 26 05:51:40 dist-test-slave-7fr8 krb5kdc[29144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401900, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:40.826699 29385 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:51:40.827193 29385 webserver.cc:460] Webserver started at http://127.24.255.2:43523/ using document root <none> and password file <none>
I20260926 05:51:40.827412 29385 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:40.827462 29385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:40.827587 29385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:40.828679 29385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "8278467cebdf4456a2dbe6da4557829e"
format_stamp: "Formatted at 2026-09-26 05:51:40 on dist-test-slave-7fr8"
server_key: "a07a752083bd026328c091be9551b493"
server_key_iv: "fc3c51e41529d2203a0b748591faf16b"
server_key_version: "encryptionkey@0"
I20260926 05:51:40.829025 29385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "8278467cebdf4456a2dbe6da4557829e"
format_stamp: "Formatted at 2026-09-26 05:51:40 on dist-test-slave-7fr8"
server_key: "a07a752083bd026328c091be9551b493"
server_key_iv: "fc3c51e41529d2203a0b748591faf16b"
server_key_version: "encryptionkey@0"
I20260926 05:51:40.830492 29385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:51:40.831347 29400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:40.831586 29385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:40.831686 29385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "8278467cebdf4456a2dbe6da4557829e"
format_stamp: "Formatted at 2026-09-26 05:51:40 on dist-test-slave-7fr8"
server_key: "a07a752083bd026328c091be9551b493"
server_key_iv: "fc3c51e41529d2203a0b748591faf16b"
server_key_version: "encryptionkey@0"
I20260926 05:51:40.831761 29385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:40.849152 29385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:40.851438 29385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:40.851585 29385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:40.851802 29385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:40.852105 29385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:40.852138 29385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:40.852169 29385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:40.852191 29385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:40.857748 29385 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:34451
I20260926 05:51:40.857789 29513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:34451 every 8 connection(s)
I20260926 05:51:40.858132 29385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:51:40 dist-test-slave-7fr8 krb5kdc[29144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401900, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:40.865075 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 29385
I20260926 05:51:40.865188 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:51:40.865383 25596 external_mini_cluster.cc:1428] Setting key 8a505f0aa997284902eabb94bf7b9eb9
I20260926 05:51:40.867976 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:40.858844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:57581 (local address 127.24.255.62:38801)
0926 05:51:40.858922 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:40.858924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:40.859151 (+ 227us) server_negotiation.cc:407] Connection header received
0926 05:51:40.860067 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:40.860069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:40.860082 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:40.860129 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:40.860813 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:40.861316 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:40.862366 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:40.862556 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:40.862611 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:40.864930 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:40.864932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:40.864933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:40.864942 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:40.866500 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:40.866897 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:40.866898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:40.866899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:40.866963 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:40.867257 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:40.867258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:40.867258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:40.867403 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:40.867468 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:40.867788 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 05:51:40.867886 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:51:40.868165 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:40.858818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38801 (local address 127.24.255.2:57581)
0926 05:51:40.859044 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:40.859076 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:40.859960 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:40.860215 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:40.860219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:40.860506 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:40.860711 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:40.860715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:40.861413 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:40.861414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:40.862288 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:40.862290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:40.862626 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:40.862627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:40.862764 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:40.863363 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:40.863373 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:40.864814 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:40.866578 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:40.866585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:40.866588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:40.866827 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:40.867055 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:40.867057 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:40.867057 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:40.867192 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:40.867500 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:40.867509 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:40.867656 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 05:51:40.867750 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 05:51:40.867922 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260926 05:51:40.868746 29514 heartbeater.cc:347] Connected to a master server at 127.24.255.62:38801
I20260926 05:51:40.868873 29514 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:40.869073 29514 heartbeater.cc:511] Master 127.24.255.62:38801 requested a full tablet report, sending...
I20260926 05:51:40.869525 29189 ts_manager.cc:194] Registered new tserver with Master: 8278467cebdf4456a2dbe6da4557829e (127.24.255.2:34451)
I20260926 05:51:40.870054 29189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:57581
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:51:40.894161 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38801
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:37071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:40.980962 29519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:40.981182 29519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:40.981206 29519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:40.982415 29519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:40.982462 29519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:40.982491 29519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:40.984058 29519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:40.984273 29519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:40.984462 29519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:40.989522 29527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:40.989574 29525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:40.989563 29524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:40.989826 29519 server_base.cc:1034] running on GCE node
I20260926 05:51:40.990165 29519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:40.990415 29519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:40.991586 29519 hybrid_clock.cc:648] HybridClock initialized: now 1790401900991573 us; error 27 us; skew 500 ppm
Sep 26 05:51:40 dist-test-slave-7fr8 krb5kdc[29144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401900, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:40.994262 29519 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:51:40.994791 29519 webserver.cc:460] Webserver started at http://127.24.255.3:46839/ using document root <none> and password file <none>
I20260926 05:51:40.995072 29519 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:40.995131 29519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:40.995265 29519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:40.996519 29519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "9f49e7674502490596a870d1869161af"
format_stamp: "Formatted at 2026-09-26 05:51:40 on dist-test-slave-7fr8"
server_key: "7ba1a873719f60edc24c2ad0beebccd0"
server_key_iv: "8670171971d7f4d7606fcea4df16ff1b"
server_key_version: "encryptionkey@0"
I20260926 05:51:40.997016 29519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "9f49e7674502490596a870d1869161af"
format_stamp: "Formatted at 2026-09-26 05:51:40 on dist-test-slave-7fr8"
server_key: "7ba1a873719f60edc24c2ad0beebccd0"
server_key_iv: "8670171971d7f4d7606fcea4df16ff1b"
server_key_version: "encryptionkey@0"
I20260926 05:51:40.998539 29519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:40.999272 29534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:40.999454 29519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:51:40.999684 29519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "9f49e7674502490596a870d1869161af"
format_stamp: "Formatted at 2026-09-26 05:51:40 on dist-test-slave-7fr8"
server_key: "7ba1a873719f60edc24c2ad0beebccd0"
server_key_iv: "8670171971d7f4d7606fcea4df16ff1b"
server_key_version: "encryptionkey@0"
I20260926 05:51:40.999768 29519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:41.018836 29519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:41.020915 29519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:41.021034 29519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:41.021247 29519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:41.021539 29519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:41.021572 29519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:41.021596 29519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:41.021618 29519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:41.027184 29519 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:37499
I20260926 05:51:41.027237 29647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:37499 every 8 connection(s)
I20260926 05:51:41.027546 29519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:51:41.029794 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 29519
I20260926 05:51:41.029886 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:51:41.030066 25596 external_mini_cluster.cc:1428] Setting key 518b82595bb54ac7e86600fa94c1e6fa
Sep 26 05:51:41 dist-test-slave-7fr8 krb5kdc[29144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401900, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:41.037649 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:41.028271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:42169 (local address 127.24.255.62:38801)
0926 05:51:41.028353 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:41.028355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:41.028569 (+ 214us) server_negotiation.cc:407] Connection header received
0926 05:51:41.029461 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:41.029462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:41.029476 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:41.029552 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:41.030393 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:41.031066 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:41.032093 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:41.032288 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:41.032353 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:41.034569 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:41.034571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:41.034571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:41.034580 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:41.036230 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:41.036652 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:41.036653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:41.036653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:41.036715 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:41.036941 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:41.036942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:41.036942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:41.037048 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:41.037094 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:41.037490 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 05:51:41.037575 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 05:51:41.037796 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:41.028221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38801 (local address 127.24.255.3:42169)
0926 05:51:41.028432 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:41.028475 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:41.029364 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:41.029614 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:41.029618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:41.030000 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:41.030311 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:41.030316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:41.031144 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:41.031145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:41.031994 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:41.031997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:41.032391 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:41.032392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:41.032527 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:41.033060 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:41.033069 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:41.034439 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:41.036323 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:41.036331 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:41.036334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:41.036580 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:41.036767 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:41.036767 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:41.036767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:41.036891 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:41.037155 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:41.037162 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:41.037303 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 05:51:41.037400 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 05:51:41.037589 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":123,"threads_started":1}
I20260926 05:51:41.038408 29648 heartbeater.cc:347] Connected to a master server at 127.24.255.62:38801
I20260926 05:51:41.038517 29648 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:41.038671 29648 heartbeater.cc:511] Master 127.24.255.62:38801 requested a full tablet report, sending...
I20260926 05:51:41.039062 29189 ts_manager.cc:194] Registered new tserver with Master: 9f49e7674502490596a870d1869161af (127.24.255.3:37499)
I20260926 05:51:41.039413 29189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:42169
I20260926 05:51:41.041960 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 05:51:41 dist-test-slave-7fr8 krb5kdc[29144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:51:41 dist-test-slave-7fr8 krb5kdc[29144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.1@KRBTEST.COM
I20260926 05:51:41.066689 29655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:41.057978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.24.255.1:39567)
0926 05:51:41.058192 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:41.058198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:41.058210 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:51:41.058234 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:41.058236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:41.058251 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:41.058314 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:41.058913 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:41.059711 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:41.060465 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:41.060795 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:41.060881 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:41.063463 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:41.063474 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:41.063474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:41.063487 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:41.065307 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:41.065823 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:41.065827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:41.065828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:41.065920 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:41.066210 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:41.066210 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:41.066211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:41.066365 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:41.066480 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:41.066535 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 05:51:41.066605 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":147,"threads_started":1}
Sep 26 05:51:41 dist-test-slave-7fr8 krb5kdc[29144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:41.075176 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:41.068232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.24.255.62:38801)
0926 05:51:41.068293 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:41.068295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:41.068303 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:41.068325 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:41.068325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:41.068333 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:41.068377 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:41.068901 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:41.069418 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:41.070176 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:41.070368 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:41.070424 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:41.072585 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:41.072587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:41.072587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:41.072595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:41.073935 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:41.074331 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:41.074332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:41.074333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:41.074405 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:41.074649 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:41.074650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:41.074651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:41.074821 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:41.074903 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:41.075034 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 05:51:41.075100 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 05:51:41.075490 29189 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49280
I20260926 05:51:41.075837 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 29251
I20260926 05:51:41.079905 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 29385
I20260926 05:51:41.083504 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 29519
I20260926 05:51:41.087519 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 29157
2026-09-26T05:51:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1826 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:41 dist-test-slave-7fr8 krb5kdc[29660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:41 dist-test-slave-7fr8 krb5kdc[29660](info): set up 2 sockets
Sep 26 05:51:41 dist-test-slave-7fr8 krb5kdc[29660](info): commencing operation
krb5kdc: starting...
W20260926 05:51:42.066643 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:42 dist-test-slave-7fr8 krb5kdc[29660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:51:42.194547 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:42999
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:42999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:42.283408 29673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:42.283656 29673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:42.283681 29673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:42.284996 29673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:42.285054 29673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:42.285068 29673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:42.285079 29673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:42.285091 29673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:42.286885 29673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:42999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:42999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:42.287164 29673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:42.287403 29673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:42.292524 29679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:42.292536 29681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:42.292526 29678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:42.292524 29673 server_base.cc:1034] running on GCE node
I20260926 05:51:42.292924 29673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:42.293292 29673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:42.294550 29673 hybrid_clock.cc:648] HybridClock initialized: now 1790401902294502 us; error 44 us; skew 500 ppm
Sep 26 05:51:42 dist-test-slave-7fr8 krb5kdc[29660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401902, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:42.297794 29673 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:51:42.298231 29673 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:42.298291 29673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:42.298426 29673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:42.299566 29673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "7014baacc3f644b9b02c31d39b260e0e"
format_stamp: "Formatted at 2026-09-26 05:51:42 on dist-test-slave-7fr8"
server_key: "291f64a11bf22acbe7a1bbcce0f1cf9e"
server_key_iv: "2b089cd568c8c4e694fb961089e89e01"
server_key_version: "encryptionkey@0"
I20260926 05:51:42.299918 29673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "7014baacc3f644b9b02c31d39b260e0e"
format_stamp: "Formatted at 2026-09-26 05:51:42 on dist-test-slave-7fr8"
server_key: "291f64a11bf22acbe7a1bbcce0f1cf9e"
server_key_iv: "2b089cd568c8c4e694fb961089e89e01"
server_key_version: "encryptionkey@0"
I20260926 05:51:42.301319 29673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:42.302425 29687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:42.302745 29673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:42.302870 29673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "7014baacc3f644b9b02c31d39b260e0e"
format_stamp: "Formatted at 2026-09-26 05:51:42 on dist-test-slave-7fr8"
server_key: "291f64a11bf22acbe7a1bbcce0f1cf9e"
server_key_iv: "2b089cd568c8c4e694fb961089e89e01"
server_key_version: "encryptionkey@0"
I20260926 05:51:42.302945 29673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:42.311656 29673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:42.313815 29673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:42.313956 29673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:42.318367 29673 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:42999
I20260926 05:51:42.318394 29739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:42999 every 8 connection(s)
I20260926 05:51:42.318694 29673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:42.319905 29740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:42.320242 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 29673
I20260926 05:51:42.320346 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:42.320537 25596 external_mini_cluster.cc:1428] Setting key 03354e8b31d800e1cd8b91e6cadbe5b4
I20260926 05:51:42.323760 29740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e: Bootstrap starting.
I20260926 05:51:42.324415 29740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:42.324688 29740 log.cc:824] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:42.325363 29740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e: No bootstrap required, opened a new log
Sep 26 05:51:42 dist-test-slave-7fr8 krb5kdc[29660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:42.327294 29740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7014baacc3f644b9b02c31d39b260e0e" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42999 } }
I20260926 05:51:42.327453 29740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:42.327486 29740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7014baacc3f644b9b02c31d39b260e0e, State: Initialized, Role: FOLLOWER
I20260926 05:51:42.327628 29740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7014baacc3f644b9b02c31d39b260e0e" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42999 } }
I20260926 05:51:42.327705 29740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:42.327739 29740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:42.327788 29740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:42.328349 29740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7014baacc3f644b9b02c31d39b260e0e" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42999 } }
I20260926 05:51:42.328460 29740 leader_election.cc:302] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7014baacc3f644b9b02c31d39b260e0e; no voters:
I20260926 05:51:42.328626 29740 leader_election.cc:288] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:42.328687 29745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:42.328835 29745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [term 1 LEADER]: Becoming Leader. State: Replica: 7014baacc3f644b9b02c31d39b260e0e, State: Running, Role: LEADER
I20260926 05:51:42.328894 29740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:42.328939 29745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7014baacc3f644b9b02c31d39b260e0e" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42999 } }
I20260926 05:51:42.329258 29745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7014baacc3f644b9b02c31d39b260e0e. Latest consensus state: current_term: 1 leader_uuid: "7014baacc3f644b9b02c31d39b260e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7014baacc3f644b9b02c31d39b260e0e" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42999 } } }
I20260926 05:51:42.329342 29745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:42.329643 29747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7014baacc3f644b9b02c31d39b260e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7014baacc3f644b9b02c31d39b260e0e" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42999 } } }
I20260926 05:51:42.329768 29747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:42.330011 29751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:42.330515 29751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:42.330859 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.321172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.24.255.62:42999)
0926 05:51:42.321340 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:42.321351 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:42.321387 (+ 36us) server_negotiation.cc:407] Connection header received
0926 05:51:42.322007 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:42.322015 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:42.322070 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:42.322243 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:42.322874 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.323762 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.324543 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.324992 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.325089 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.327326 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:42.327331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:42.327331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:42.327343 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:42.329111 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:42.329541 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:42.329544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:42.329545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:42.329647 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:42.329872 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:42.329873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:42.329874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:42.330231 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:42.330396 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:42.330556 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 05:51:42.330700 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260926 05:51:42.332113 29751 catalog_manager.cc:1347] Generated new cluster ID: a19d5086d1e74b36b88431c6c20cb9ff
I20260926 05:51:42.332168 29751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:42.340013 29751 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:42.340664 29751 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:42.344445 29751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e: Generated new TSK 0
I20260926 05:51:42.344666 29751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:51:42.377956 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42999
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:42.468364 29766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:42.468622 29766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:42.468649 29766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:42.469952 29766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:42.470009 29766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:42.470039 29766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:42.471740 29766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:42.471987 29766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:42.472229 29766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:42.477705 29772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:42.477807 29774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:42.477789 29771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:42.478205 29766 server_base.cc:1034] running on GCE node
I20260926 05:51:42.478497 29766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:42.478844 29766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:42.480058 29766 hybrid_clock.cc:648] HybridClock initialized: now 1790401902480022 us; error 63 us; skew 500 ppm
Sep 26 05:51:42 dist-test-slave-7fr8 krb5kdc[29660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401902, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:42.483559 29766 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:51:42.483983 29766 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:42.484040 29766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:42.484167 29766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:42.485435 29766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "110ec90652db405f99d3449c1655d9ed"
format_stamp: "Formatted at 2026-09-26 05:51:42 on dist-test-slave-7fr8"
server_key: "9999ac0fab981aad34d4993a0bb63aa7"
server_key_iv: "354ed99233e60b1565512b6a75e30752"
server_key_version: "encryptionkey@0"
I20260926 05:51:42.485783 29766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "110ec90652db405f99d3449c1655d9ed"
format_stamp: "Formatted at 2026-09-26 05:51:42 on dist-test-slave-7fr8"
server_key: "9999ac0fab981aad34d4993a0bb63aa7"
server_key_iv: "354ed99233e60b1565512b6a75e30752"
server_key_version: "encryptionkey@0"
I20260926 05:51:42.487221 29766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:51:42.488241 29780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:42.488510 29766 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:42.488611 29766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "110ec90652db405f99d3449c1655d9ed"
format_stamp: "Formatted at 2026-09-26 05:51:42 on dist-test-slave-7fr8"
server_key: "9999ac0fab981aad34d4993a0bb63aa7"
server_key_iv: "354ed99233e60b1565512b6a75e30752"
server_key_version: "encryptionkey@0"
I20260926 05:51:42.488695 29766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:42.500393 29766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:42.502722 29766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:42.502861 29766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:42.503104 29766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:42.503404 29766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:42.503443 29766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:42.503471 29766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:42.503486 29766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:42.509683 29766 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:40447
I20260926 05:51:42.509725 29893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:40447 every 8 connection(s)
I20260926 05:51:42.510001 29766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:51:42.513206 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 29766
I20260926 05:51:42.513321 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:51:42.513517 25596 external_mini_cluster.cc:1428] Setting key b3b3862581b230871efeb310219c108d
Sep 26 05:51:42 dist-test-slave-7fr8 krb5kdc[29660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401902, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:42.520509 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.510919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:34143 (local address 127.24.255.62:42999)
0926 05:51:42.511007 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:42.511010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:42.511161 (+ 151us) server_negotiation.cc:407] Connection header received
0926 05:51:42.511642 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:42.511644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:42.511654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:42.511719 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:42.512802 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.513594 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.514834 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.515112 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.515197 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.517675 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:42.517677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:42.517677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:42.517685 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:42.519033 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:42.519442 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:42.519443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:42.519443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:42.519503 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:42.519732 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:42.519733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:42.519734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:42.519852 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:42.519907 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:42.520317 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 05:51:42.520411 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 05:51:42.520748 29896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.510837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:42999 (local address 127.24.255.1:34143)
0926 05:51:42.511002 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:42.511047 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:42.511528 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:42.511879 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:42.511883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:42.512158 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:42.512670 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.512676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.513694 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.513695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.514722 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.514726 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.515250 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.515251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.515373 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.515932 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:42.515943 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:42.517561 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:42.519123 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:42.519132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:42.519135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:42.519369 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:42.519561 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:42.519562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:42.519563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:42.519675 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:42.519908 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:42.519915 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:42.520144 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 05:51:42.520273 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 05:51:42.520519 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260926 05:51:42.521275 29894 heartbeater.cc:347] Connected to a master server at 127.24.255.62:42999
I20260926 05:51:42.521359 29894 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
I20260926 05:51:42.521515 29894 heartbeater.cc:511] Master 127.24.255.62:42999 requested a full tablet report, sending...
I20260926 05:51:42.522197 29704 ts_manager.cc:194] Registered new tserver with Master: 110ec90652db405f99d3449c1655d9ed (127.24.255.1:40447)
I20260926 05:51:42.522843 29704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:34143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:51:42.541663 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42999
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:42.625285 29899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:42.625530 29899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:42.625566 29899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:42.626952 29899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:42.627007 29899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:42.627045 29899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:42.628805 29899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:42.629078 29899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:42.629316 29899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:42.634711 29907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:42.634696 29904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:42.634696 29905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:42.634917 29899 server_base.cc:1034] running on GCE node
I20260926 05:51:42.635223 29899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:42.635471 29899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:42.636653 29899 hybrid_clock.cc:648] HybridClock initialized: now 1790401902636632 us; error 38 us; skew 500 ppm
Sep 26 05:51:42 dist-test-slave-7fr8 krb5kdc[29660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401902, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:42.639217 29899 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:51:42.639578 29899 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:42.639629 29899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:42.639722 29899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:42.640760 29899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "217a6ea5d6454cc99baa324ad4f263f4"
format_stamp: "Formatted at 2026-09-26 05:51:42 on dist-test-slave-7fr8"
server_key: "e91efe100bcd0343c6aff8215d423c3a"
server_key_iv: "ebfb09295d7e96ea86a821d461f79950"
server_key_version: "encryptionkey@0"
I20260926 05:51:42.641214 29899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "217a6ea5d6454cc99baa324ad4f263f4"
format_stamp: "Formatted at 2026-09-26 05:51:42 on dist-test-slave-7fr8"
server_key: "e91efe100bcd0343c6aff8215d423c3a"
server_key_iv: "ebfb09295d7e96ea86a821d461f79950"
server_key_version: "encryptionkey@0"
I20260926 05:51:42.642700 29899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:51:42.643623 29913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:42.643842 29899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:42.643953 29899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "217a6ea5d6454cc99baa324ad4f263f4"
format_stamp: "Formatted at 2026-09-26 05:51:42 on dist-test-slave-7fr8"
server_key: "e91efe100bcd0343c6aff8215d423c3a"
server_key_iv: "ebfb09295d7e96ea86a821d461f79950"
server_key_version: "encryptionkey@0"
I20260926 05:51:42.644018 29899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:42.656765 29899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:42.659227 29899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:42.659412 29899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:42.659618 29899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:42.659932 29899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:42.659972 29899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:42.659994 29899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:42.660010 29899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:42.665723 29899 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:35593
I20260926 05:51:42.665782 30026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:35593 every 8 connection(s)
I20260926 05:51:42.666040 29899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:51:42.667369 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 29899
I20260926 05:51:42.667559 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:51:42.667770 25596 external_mini_cluster.cc:1428] Setting key c334d43a21e72969ec85d20b77681610
Sep 26 05:51:42 dist-test-slave-7fr8 krb5kdc[29660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401902, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
I20260926 05:51:42.678546 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.666833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:38215 (local address 127.24.255.62:42999)
0926 05:51:42.666922 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:42.666925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:42.667064 (+ 139us) server_negotiation.cc:407] Connection header received
0926 05:51:42.667876 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:42.667878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:42.667891 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:42.668003 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:42.669203 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.669714 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.670879 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.671170 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.671247 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.674311 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:42.674314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:42.674314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:42.674322 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:42.676171 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:42.677303 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:42.677305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:42.677305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:42.677394 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:42.677757 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:42.677759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:42.677759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:42.677893 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:42.677959 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:42.678338 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 05:51:42.678443 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 05:51:42.678810 30029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.666773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:42999 (local address 127.24.255.2:38215)
0926 05:51:42.666925 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:42.666971 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:42.667743 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:42.668109 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:42.668113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:42.668562 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:42.669094 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.669099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.669826 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.669828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.670804 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.670807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.671279 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.671280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.671401 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.672068 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:42.672086 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:42.674149 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:42.676276 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:42.676465 (+ 189us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:42.676471 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:42.677149 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:42.677494 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:42.677495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:42.677495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:42.677685 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:42.677966 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:42.677973 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:42.678165 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 05:51:42.678306 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 05:51:42.678569 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
I20260926 05:51:42.679519 30027 heartbeater.cc:347] Connected to a master server at 127.24.255.62:42999
I20260926 05:51:42.679646 30027 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:42.679819 30027 heartbeater.cc:511] Master 127.24.255.62:42999 requested a full tablet report, sending...
I20260926 05:51:42.680437 29704 ts_manager.cc:194] Registered new tserver with Master: 217a6ea5d6454cc99baa324ad4f263f4 (127.24.255.2:35593)
I20260926 05:51:42.681073 29704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:38215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:51:42.698621 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42999
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:42.783847 30032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:42.784067 30032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:42.784104 30032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:42.785337 30032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:42.785391 30032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:42.785430 30032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:42.787067 30032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:42.787328 30032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:42.787554 30032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:42.792739 30038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:42.792742 30040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:42.792829 30032 server_base.cc:1034] running on GCE node
W20260926 05:51:42.792749 30037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:42.793165 30032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:42.793431 30032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:42.794620 30032 hybrid_clock.cc:648] HybridClock initialized: now 1790401902794595 us; error 37 us; skew 500 ppm
Sep 26 05:51:42 dist-test-slave-7fr8 krb5kdc[29660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401902, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:42.797268 30032 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:51:42.797628 30032 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:42.797680 30032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:42.797806 30032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:42.799166 30032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "e73ed8f927264169bc52841646d8f2ad"
format_stamp: "Formatted at 2026-09-26 05:51:42 on dist-test-slave-7fr8"
server_key: "48f7237a4b9117c601f5985df1e1b9a9"
server_key_iv: "edc5dc2ae22dfb74694343fa9f36a4d4"
server_key_version: "encryptionkey@0"
I20260926 05:51:42.799538 30032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "e73ed8f927264169bc52841646d8f2ad"
format_stamp: "Formatted at 2026-09-26 05:51:42 on dist-test-slave-7fr8"
server_key: "48f7237a4b9117c601f5985df1e1b9a9"
server_key_iv: "edc5dc2ae22dfb74694343fa9f36a4d4"
server_key_version: "encryptionkey@0"
I20260926 05:51:42.800874 30032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:42.801710 30046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:42.801899 30032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:42.801985 30032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "e73ed8f927264169bc52841646d8f2ad"
format_stamp: "Formatted at 2026-09-26 05:51:42 on dist-test-slave-7fr8"
server_key: "48f7237a4b9117c601f5985df1e1b9a9"
server_key_iv: "edc5dc2ae22dfb74694343fa9f36a4d4"
server_key_version: "encryptionkey@0"
I20260926 05:51:42.802050 30032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:42.820817 30032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:42.822968 30032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:42.823122 30032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:42.823519 30032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:42.823920 30032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:42.823959 30032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:42.823992 30032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:42.824015 30032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:42.830322 30032 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:39591
I20260926 05:51:42.830361 30159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:39591 every 8 connection(s)
I20260926 05:51:42.830654 30032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:51:42.834167 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 30032
I20260926 05:51:42.834338 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:51:42.834546 25596 external_mini_cluster.cc:1428] Setting key 62dd095061bb3dec2bdfb277dbcb9383
Sep 26 05:51:42 dist-test-slave-7fr8 krb5kdc[29660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401902, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:42.841012 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.831486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:39941 (local address 127.24.255.62:42999)
0926 05:51:42.831545 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:42.831548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:42.831766 (+ 218us) server_negotiation.cc:407] Connection header received
0926 05:51:42.832264 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:42.832265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:42.832275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:42.832324 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:42.833292 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.833834 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.835128 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.835409 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.835483 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.837689 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:42.837691 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:42.837691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:42.837698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:42.839227 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:42.839701 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:42.839702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:42.839702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:42.839761 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:42.840240 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:42.840241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:42.840242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:42.840372 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:42.840489 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:42.840773 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 05:51:42.840935 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:51:42.841544 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.831491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:42999 (local address 127.24.255.3:39941)
0926 05:51:42.831631 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:42.831667 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:42.832171 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:42.832385 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:42.832388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:42.832654 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:42.833159 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.833165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.833976 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.833977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.835048 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.835052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.835494 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.835495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.835611 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.836095 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:42.836103 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:42.837579 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:42.839358 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:42.839365 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:42.839367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:42.839610 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:42.839934 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:42.839935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:42.839935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:42.840143 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:42.840468 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:42.840476 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:42.840626 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 05:51:42.840730 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 05:51:42.841360 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260926 05:51:42.842383 30160 heartbeater.cc:347] Connected to a master server at 127.24.255.62:42999
I20260926 05:51:42.842481 30160 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:42.842626 30160 heartbeater.cc:511] Master 127.24.255.62:42999 requested a full tablet report, sending...
I20260926 05:51:42.843075 29704 ts_manager.cc:194] Registered new tserver with Master: e73ed8f927264169bc52841646d8f2ad (127.24.255.3:39591)
I20260926 05:51:42.843477 29704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:39941
I20260926 05:51:42.846474 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:51:42.853576 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.847742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.24.255.62:42999)
0926 05:51:42.847805 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:42.847807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:42.847876 (+ 69us) server_negotiation.cc:407] Connection header received
0926 05:51:42.847942 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:42.847943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:42.847953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:42.848034 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:42.848667 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.849160 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.849852 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.850039 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.850115 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.851239 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:42.851241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:42.851241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:42.851248 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:42.852513 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:42.852838 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:42.852839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:42.852840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:42.852890 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:42.853050 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:42.853051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:42.853051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:42.853205 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:42.853295 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:42.853402 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 05:51:42.853480 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:51:42.855526 29704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:51:42.857137 29704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:42.864204 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.860085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58626 (local address 127.24.255.3:39591)
0926 05:51:42.860332 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:42.860337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:42.860348 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:51:42.860389 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:42.860390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:42.860403 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:42.860471 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:42.860667 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.861516 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.863090 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.863796 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.863871 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.863933 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:42.864072 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 05:51:42.864117 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260926 05:51:42.864233 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.859904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:40447 (local address 127.0.0.1:42110)
0926 05:51:42.860244 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:42.860253 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:42.860291 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:42.860747 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:42.860748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:42.860751 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:42.860814 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.860816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.861779 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.861780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.862961 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.862963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.863970 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.863971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.864051 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.864056 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:42.864105 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 05:51:42.864146 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":108,"threads_started":1}
I20260926 05:51:42.864236 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.859901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:39591 (local address 127.0.0.1:58626)
0926 05:51:42.860087 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:42.860127 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:42.860193 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:42.860517 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:42.860519 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:42.860521 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:42.860583 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.860584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.861607 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.861608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.863016 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.863017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.863872 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.863872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.863956 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.863971 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 05:51:42.864047 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 05:51:42.864105 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":55,"threads_started":1}
I20260926 05:51:42.864562 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.860129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.24.255.1:40447)
0926 05:51:42.860589 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:42.860593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:42.860599 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:51:42.860627 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:42.860629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:42.860639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:42.860892 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:42.860971 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.861690 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.863053 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.863796 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.863872 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.863941 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:42.864437 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 05:51:42.864484 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":81,"threads_started":1}
I20260926 05:51:42.865144 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.859904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:35593 (local address 127.0.0.1:50850)
0926 05:51:42.860345 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:42.860351 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:42.860374 (+ 23us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:42.861963 (+ 1589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:42.861964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:42.861966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:42.862030 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.862032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.862946 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.862947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.863723 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.863725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.864891 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.864892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.864968 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.864973 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:42.865033 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:51:42.865071 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":107,"threads_started":1}
I20260926 05:51:42.865252 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.860193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.24.255.2:35593)
0926 05:51:42.861823 (+ 1630us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:42.861826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:42.861833 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:42.861861 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:42.861863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:42.861875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:42.861937 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:42.862127 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.862845 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.863813 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.864800 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.864872 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.864928 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:42.865091 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 05:51:42.865163 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1598,"thread_start_us":62,"threads_started":1}
I20260926 05:51:42.865156 30094 tablet_service.cc:1467] Processing CreateTablet for tablet 85e9e2d5e6b74e808d368ebfd2c5b36a (DEFAULT_TABLE table=test-table [id=fd41362dad454ed4a30fe120d94f1d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:42.865536 30094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e9e2d5e6b74e808d368ebfd2c5b36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:42.865955 29961 tablet_service.cc:1467] Processing CreateTablet for tablet 85e9e2d5e6b74e808d368ebfd2c5b36a (DEFAULT_TABLE table=test-table [id=fd41362dad454ed4a30fe120d94f1d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:42.866282 29961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e9e2d5e6b74e808d368ebfd2c5b36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:42.866652 29828 tablet_service.cc:1467] Processing CreateTablet for tablet 85e9e2d5e6b74e808d368ebfd2c5b36a (DEFAULT_TABLE table=test-table [id=fd41362dad454ed4a30fe120d94f1d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:42.866878 29828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e9e2d5e6b74e808d368ebfd2c5b36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:42.869022 30177 tablet_bootstrap.cc:492] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad: Bootstrap starting.
I20260926 05:51:42.869665 30178 tablet_bootstrap.cc:492] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed: Bootstrap starting.
I20260926 05:51:42.869688 30177 tablet_bootstrap.cc:654] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:42.870260 30179 tablet_bootstrap.cc:492] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4: Bootstrap starting.
I20260926 05:51:42.870311 30178 tablet_bootstrap.cc:654] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:42.870338 30177 log.cc:824] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:42.870796 30178 log.cc:824] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:42.870878 30179 tablet_bootstrap.cc:654] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:42.871137 30177 tablet_bootstrap.cc:492] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad: No bootstrap required, opened a new log
I20260926 05:51:42.871219 30177 ts_tablet_manager.cc:1397] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:51:42.871409 30179 log.cc:824] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:42.871546 30178 tablet_bootstrap.cc:492] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed: No bootstrap required, opened a new log
I20260926 05:51:42.871608 30178 ts_tablet_manager.cc:1397] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:51:42.872113 30179 tablet_bootstrap.cc:492] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4: No bootstrap required, opened a new log
I20260926 05:51:42.872171 30179 ts_tablet_manager.cc:1397] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:51:42.873250 30177 raft_consensus.cc:348] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73ed8f927264169bc52841646d8f2ad" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39591 } } peers { permanent_uuid: "217a6ea5d6454cc99baa324ad4f263f4" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35593 } } peers { permanent_uuid: "110ec90652db405f99d3449c1655d9ed" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40447 } }
I20260926 05:51:42.873430 30177 raft_consensus.cc:374] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:42.873464 30177 raft_consensus.cc:729] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e73ed8f927264169bc52841646d8f2ad, State: Initialized, Role: FOLLOWER
I20260926 05:51:42.873484 30178 raft_consensus.cc:348] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73ed8f927264169bc52841646d8f2ad" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39591 } } peers { permanent_uuid: "217a6ea5d6454cc99baa324ad4f263f4" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35593 } } peers { permanent_uuid: "110ec90652db405f99d3449c1655d9ed" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40447 } }
I20260926 05:51:42.873631 30178 raft_consensus.cc:374] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:42.873608 30177 consensus_queue.cc:260] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73ed8f927264169bc52841646d8f2ad" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39591 } } peers { permanent_uuid: "217a6ea5d6454cc99baa324ad4f263f4" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35593 } } peers { permanent_uuid: "110ec90652db405f99d3449c1655d9ed" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40447 } }
I20260926 05:51:42.873669 30178 raft_consensus.cc:729] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110ec90652db405f99d3449c1655d9ed, State: Initialized, Role: FOLLOWER
I20260926 05:51:42.873785 30178 consensus_queue.cc:260] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73ed8f927264169bc52841646d8f2ad" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39591 } } peers { permanent_uuid: "217a6ea5d6454cc99baa324ad4f263f4" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35593 } } peers { permanent_uuid: "110ec90652db405f99d3449c1655d9ed" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40447 } }
I20260926 05:51:42.873878 30177 ts_tablet_manager.cc:1428] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:51:42.873822 30179 raft_consensus.cc:348] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73ed8f927264169bc52841646d8f2ad" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39591 } } peers { permanent_uuid: "217a6ea5d6454cc99baa324ad4f263f4" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35593 } } peers { permanent_uuid: "110ec90652db405f99d3449c1655d9ed" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40447 } }
I20260926 05:51:42.873970 30179 raft_consensus.cc:374] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:42.874010 30179 raft_consensus.cc:729] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217a6ea5d6454cc99baa324ad4f263f4, State: Initialized, Role: FOLLOWER
I20260926 05:51:42.874024 30178 ts_tablet_manager.cc:1428] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:51:42.874154 29894 heartbeater.cc:503] Master 127.24.255.62:42999 was elected leader, sending a full tablet report...
I20260926 05:51:42.874142 30179 consensus_queue.cc:260] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73ed8f927264169bc52841646d8f2ad" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39591 } } peers { permanent_uuid: "217a6ea5d6454cc99baa324ad4f263f4" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35593 } } peers { permanent_uuid: "110ec90652db405f99d3449c1655d9ed" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40447 } }
I20260926 05:51:42.874226 30160 heartbeater.cc:503] Master 127.24.255.62:42999 was elected leader, sending a full tablet report...
I20260926 05:51:42.874392 30179 ts_tablet_manager.cc:1428] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:51:42.874429 30027 heartbeater.cc:503] Master 127.24.255.62:42999 was elected leader, sending a full tablet report...
I20260926 05:51:42.884752 30185 raft_consensus.cc:482] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:42.885003 30185 raft_consensus.cc:504] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73ed8f927264169bc52841646d8f2ad" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39591 } } peers { permanent_uuid: "217a6ea5d6454cc99baa324ad4f263f4" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35593 } } peers { permanent_uuid: "110ec90652db405f99d3449c1655d9ed" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40447 } }
I20260926 05:51:42.885373 30185 leader_election.cc:288] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e73ed8f927264169bc52841646d8f2ad (127.24.255.3:39591), 110ec90652db405f99d3449c1655d9ed (127.24.255.1:40447)
I20260926 05:51:42.888427 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.885692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:59313 (local address 127.24.255.1:40447)
0926 05:51:42.885747 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:42.885749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:42.885858 (+ 109us) server_negotiation.cc:407] Connection header received
0926 05:51:42.885916 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:42.885917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:42.885926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:42.885983 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:42.886130 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.886594 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.887690 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.888090 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.888159 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.888168 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:42.888320 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 05:51:42.888358 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:51:42.888439 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.885646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:35279 (local address 127.24.255.3:39591)
0926 05:51:42.885700 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:42.885701 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:42.885710 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:42.885738 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:42.885739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:42.885748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:42.885797 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:42.885941 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.886493 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.887687 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.888093 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.888159 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.888181 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:42.888329 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 05:51:42.888368 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:51:42.888453 30029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.885558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:39591 (local address 127.24.255.2:35279)
0926 05:51:42.885604 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:42.885617 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:42.885660 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:42.885800 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:42.885801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:42.885802 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:42.885877 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.885879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.886569 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.886570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.887570 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.887572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.888170 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.888171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.888267 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.888270 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:51:42.888316 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:51:42.888360 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":21}
I20260926 05:51:42.888459 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.885611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:40447 (local address 127.24.255.2:59313)
0926 05:51:42.885784 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:42.885794 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:42.885854 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:42.885988 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:42.885988 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:42.885989 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:42.886055 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.886056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.886669 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.886670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.887581 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:42.887584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.888170 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.888171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:42.888258 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.888262 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:51:42.888305 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 05:51:42.888347 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":3,"thread_start_us":126,"threads_started":1}
I20260926 05:51:42.888813 30114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e9e2d5e6b74e808d368ebfd2c5b36a" candidate_uuid: "217a6ea5d6454cc99baa324ad4f263f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e73ed8f927264169bc52841646d8f2ad" is_pre_election: true
I20260926 05:51:42.888995 30114 raft_consensus.cc:2454] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 217a6ea5d6454cc99baa324ad4f263f4 in term 0.
I20260926 05:51:42.889040 29848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e9e2d5e6b74e808d368ebfd2c5b36a" candidate_uuid: "217a6ea5d6454cc99baa324ad4f263f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110ec90652db405f99d3449c1655d9ed" is_pre_election: true
I20260926 05:51:42.889173 29848 raft_consensus.cc:2454] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 217a6ea5d6454cc99baa324ad4f263f4 in term 0.
I20260926 05:51:42.889247 29916 leader_election.cc:302] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 217a6ea5d6454cc99baa324ad4f263f4, e73ed8f927264169bc52841646d8f2ad; no voters:
I20260926 05:51:42.889389 30185 raft_consensus.cc:2781] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:42.889474 30185 raft_consensus.cc:482] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:42.889518 30185 raft_consensus.cc:3037] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:42.890223 30185 raft_consensus.cc:504] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73ed8f927264169bc52841646d8f2ad" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39591 } } peers { permanent_uuid: "217a6ea5d6454cc99baa324ad4f263f4" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35593 } } peers { permanent_uuid: "110ec90652db405f99d3449c1655d9ed" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40447 } }
I20260926 05:51:42.890344 30185 leader_election.cc:288] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [CANDIDATE]: Term 1 election: Requested vote from peers e73ed8f927264169bc52841646d8f2ad (127.24.255.3:39591), 110ec90652db405f99d3449c1655d9ed (127.24.255.1:40447)
I20260926 05:51:42.890517 29848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e9e2d5e6b74e808d368ebfd2c5b36a" candidate_uuid: "217a6ea5d6454cc99baa324ad4f263f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110ec90652db405f99d3449c1655d9ed"
I20260926 05:51:42.890563 30114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e9e2d5e6b74e808d368ebfd2c5b36a" candidate_uuid: "217a6ea5d6454cc99baa324ad4f263f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e73ed8f927264169bc52841646d8f2ad"
I20260926 05:51:42.890610 29848 raft_consensus.cc:3037] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:42.890630 30114 raft_consensus.cc:3037] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:42.891358 30114 raft_consensus.cc:2454] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 217a6ea5d6454cc99baa324ad4f263f4 in term 1.
I20260926 05:51:42.891366 29848 raft_consensus.cc:2454] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 217a6ea5d6454cc99baa324ad4f263f4 in term 1.
I20260926 05:51:42.891526 29915 leader_election.cc:302] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110ec90652db405f99d3449c1655d9ed, 217a6ea5d6454cc99baa324ad4f263f4; no voters:
I20260926 05:51:42.891625 30185 raft_consensus.cc:2781] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:42.891803 30185 raft_consensus.cc:686] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [term 1 LEADER]: Becoming Leader. State: Replica: 217a6ea5d6454cc99baa324ad4f263f4, State: Running, Role: LEADER
I20260926 05:51:42.891912 30185 consensus_queue.cc:237] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73ed8f927264169bc52841646d8f2ad" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39591 } } peers { permanent_uuid: "217a6ea5d6454cc99baa324ad4f263f4" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35593 } } peers { permanent_uuid: "110ec90652db405f99d3449c1655d9ed" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40447 } }
I20260926 05:51:42.892750 29703 catalog_manager.cc:5495] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 217a6ea5d6454cc99baa324ad4f263f4 (127.24.255.2). New cstate: current_term: 1 leader_uuid: "217a6ea5d6454cc99baa324ad4f263f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73ed8f927264169bc52841646d8f2ad" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "217a6ea5d6454cc99baa324ad4f263f4" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "110ec90652db405f99d3449c1655d9ed" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40447 } health_report { overall_health: UNKNOWN } } }
I20260926 05:51:42.903198 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.900377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.24.255.2:35593)
0926 05:51:42.900443 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:42.900445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:42.900454 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:42.900544 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:42.900545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:42.900561 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:42.900623 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:42.900749 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.901528 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.902374 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:42.902619 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:42.902680 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:42.902851 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:42.902961 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:42.903033 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:42.903071 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 05:51:42.903093 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:51:42.905547 29848 raft_consensus.cc:1261] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 110ec90652db405f99d3449c1655d9ed [term 1 FOLLOWER]: Refusing update from remote peer 217a6ea5d6454cc99baa324ad4f263f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:42.905573 30114 raft_consensus.cc:1261] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad [term 1 FOLLOWER]: Refusing update from remote peer 217a6ea5d6454cc99baa324ad4f263f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:42.905941 30187 consensus_queue.cc:1035] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e73ed8f927264169bc52841646d8f2ad" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:42.906162 30185 consensus_queue.cc:1035] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "110ec90652db405f99d3449c1655d9ed" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:42.908643 30190 mvcc.cc:205] Tried to move back new op lower bound from 7333486194297184256 to 7333486194245935104. Current Snapshot: MvccSnapshot[applied={T|T < 7333486194297184256}]
I20260926 05:51:42.909050 30191 mvcc.cc:205] Tried to move back new op lower bound from 7333486194297184256 to 7333486194245935104. Current Snapshot: MvccSnapshot[applied={T|T < 7333486194297184256}]
I20260926 05:51:42.909595 30192 mvcc.cc:205] Tried to move back new op lower bound from 7333486194297184256 to 7333486194245935104. Current Snapshot: MvccSnapshot[applied={T|T < 7333486194297184256}]
I20260926 05:51:42.911139 29704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:42.911237 29704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:42.912910 29704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7014baacc3f644b9b02c31d39b260e0e: Sending DeleteTablet for 3 replicas of tablet 85e9e2d5e6b74e808d368ebfd2c5b36a
I20260926 05:51:42.913409 29961 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e9e2d5e6b74e808d368ebfd2c5b36a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:42 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:50850
I20260926 05:51:42.913488 29828 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e9e2d5e6b74e808d368ebfd2c5b36a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:42 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:42110
I20260926 05:51:42.913678 30200 tablet_replica.cc:330] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4: stopping tablet replica
I20260926 05:51:42.913808 30200 raft_consensus.cc:2229] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:51:42.913831 30094 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e9e2d5e6b74e808d368ebfd2c5b36a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:42 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:58626
I20260926 05:51:42.913942 30200 raft_consensus.cc:2258] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:42.914552 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 29766
I20260926 05:51:42.914609 30200 ts_tablet_manager.cc:1905] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:42.916447 30200 ts_tablet_manager.cc:1918] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:42.916532 30200 log.cc:1196] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/85e9e2d5e6b74e808d368ebfd2c5b36a
I20260926 05:51:42.916774 30200 ts_tablet_manager.cc:1939] T 85e9e2d5e6b74e808d368ebfd2c5b36a P 217a6ea5d6454cc99baa324ad4f263f4: Deleting consensus metadata
I20260926 05:51:42.917019 30202 tablet_replica.cc:330] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad: stopping tablet replica
I20260926 05:51:42.917143 30202 raft_consensus.cc:2229] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:42.917248 30202 raft_consensus.cc:2258] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:42.918639 29688 catalog_manager.cc:4841] TS 217a6ea5d6454cc99baa324ad4f263f4 (127.24.255.2:35593): tablet 85e9e2d5e6b74e808d368ebfd2c5b36a (table test-table [id=fd41362dad454ed4a30fe120d94f1d7d]) successfully deleted
I20260926 05:51:42.919301 30202 ts_tablet_manager.cc:1905] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:42.921593 30202 ts_tablet_manager.cc:1918] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:42.921739 30202 log.cc:1196] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/85e9e2d5e6b74e808d368ebfd2c5b36a
I20260926 05:51:42.922072 30202 ts_tablet_manager.cc:1939] T 85e9e2d5e6b74e808d368ebfd2c5b36a P e73ed8f927264169bc52841646d8f2ad: Deleting consensus metadata
I20260926 05:51:42.922945 29690 catalog_manager.cc:4841] TS e73ed8f927264169bc52841646d8f2ad (127.24.255.3:39591): tablet 85e9e2d5e6b74e808d368ebfd2c5b36a (table test-table [id=fd41362dad454ed4a30fe120d94f1d7d]) successfully deleted
W20260926 05:51:42.923094 29689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:40447 (error 108)
I20260926 05:51:42.923745 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 29899
I20260926 05:51:42.923942 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:42.923761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:40447 (local address 127.0.0.1:42118)
0926 05:51:42.923847 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:42.923907 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 05:51:42.925511 29689 catalog_manager.cc:4584] TS 110ec90652db405f99d3449c1655d9ed (127.24.255.1:40447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85e9e2d5e6b74e808d368ebfd2c5b36a: Network error: Client connection negotiation failed: client connection to 127.24.255.1:40447: connect: Connection refused (error 111)
I20260926 05:51:42.929432 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 30032
I20260926 05:51:42.933838 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 29673
2026-09-26T05:51:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1846 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T05:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:42Z Disabled control of system clock
I20260926 05:51:42.962040 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:43879
--webserver_interface=127.24.255.62
--webserver_port=0
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:43879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 05:51:43.047165 30207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:43.047400 30207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:43.047425 30207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:43.048696 30207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:43.048761 30207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:43.048772 30207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:43.048784 30207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:43.048803 30207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:43.050313 30207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:43879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:43879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:43.050547 30207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:43.050823 30207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:51:43.056447 30207 server_base.cc:1034] running on GCE node
W20260926 05:51:43.056470 30213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:43.056710 30212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:43.056547 30215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:43.057075 30207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:43.057397 30207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:43.058585 30207 hybrid_clock.cc:648] HybridClock initialized: now 1790401903058560 us; error 43 us; skew 500 ppm
I20260926 05:51:43.059968 30207 webserver.cc:460] Webserver started at http://127.24.255.62:37613/ using document root <none> and password file <none>
I20260926 05:51:43.060214 30207 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:43.060264 30207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:43.060376 30207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:43.061501 30207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "d1a7b67045ae4638b987c0320839906b"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "4e3cdcd9965f5d6323347b41657fb7ed"
server_key_iv: "fc7447937dc081469d90f9673d611e45"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.061829 30207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "d1a7b67045ae4638b987c0320839906b"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "4e3cdcd9965f5d6323347b41657fb7ed"
server_key_iv: "fc7447937dc081469d90f9673d611e45"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.063256 30207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:43.064306 30221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:43.064644 30207 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:43.064734 30207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "d1a7b67045ae4638b987c0320839906b"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "4e3cdcd9965f5d6323347b41657fb7ed"
server_key_iv: "fc7447937dc081469d90f9673d611e45"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.064817 30207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:43.068432 30207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:43.068751 30207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:43.068869 30207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:43.072832 30207 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:43879
I20260926 05:51:43.072921 30273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:43879 every 8 connection(s)
I20260926 05:51:43.073238 30207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:43.073963 30274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:43.076977 30274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b: Bootstrap starting.
I20260926 05:51:43.077124 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 30207
I20260926 05:51:43.077206 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:43.077387 25596 external_mini_cluster.cc:1428] Setting key 6416f6f3bc757749091e516b4f559dc7
I20260926 05:51:43.077728 30274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:43.078053 30274 log.cc:824] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:43.078819 30274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b: No bootstrap required, opened a new log
I20260926 05:51:43.080344 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.077993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51622 (local address 127.24.255.62:43879)
0926 05:51:43.078234 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:43.078243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:43.078279 (+ 36us) server_negotiation.cc:407] Connection header received
0926 05:51:43.078800 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:43.078808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:43.078850 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:43.079016 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:43.079863 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:43.079866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:43.079866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:43.079872 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:43.079917 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:43.079929 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:43.080012 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:43.080065 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 05:51:43.080128 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":118,"threads_started":1}
I20260926 05:51:43.081666 30274 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a7b67045ae4638b987c0320839906b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 43879 } }
I20260926 05:51:43.081831 30274 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:43.081856 30274 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a7b67045ae4638b987c0320839906b, State: Initialized, Role: FOLLOWER
I20260926 05:51:43.081969 30274 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a7b67045ae4638b987c0320839906b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 43879 } }
I20260926 05:51:43.082048 30274 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:43.082069 30274 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:43.082134 30274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:43.083050 30274 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a7b67045ae4638b987c0320839906b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 43879 } }
I20260926 05:51:43.083149 30274 leader_election.cc:302] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a7b67045ae4638b987c0320839906b; no voters:
I20260926 05:51:43.083281 30274 leader_election.cc:288] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:43.083485 30279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:43.083531 30274 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:43.083753 30279 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [term 1 LEADER]: Becoming Leader. State: Replica: d1a7b67045ae4638b987c0320839906b, State: Running, Role: LEADER
I20260926 05:51:43.083911 30279 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a7b67045ae4638b987c0320839906b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 43879 } }
I20260926 05:51:43.084641 30281 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a7b67045ae4638b987c0320839906b. Latest consensus state: current_term: 1 leader_uuid: "d1a7b67045ae4638b987c0320839906b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a7b67045ae4638b987c0320839906b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 43879 } } }
I20260926 05:51:43.084774 30281 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:43.084824 30280 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1a7b67045ae4638b987c0320839906b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a7b67045ae4638b987c0320839906b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 43879 } } }
I20260926 05:51:43.084946 30280 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:43.085142 30292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:43.086007 30292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:43.088192 30292 catalog_manager.cc:1347] Generated new cluster ID: ca3dc0745e4443c5ac8275b785071b9d
I20260926 05:51:43.088272 30292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:43.112902 30292 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:43.113094 30292 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:43.119097 30292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b: Generated new TSK 0
I20260926 05:51:43.119334 30292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:43.121569 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:43879
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 05:51:43.208178 30298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:43.208397 30298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:43.208419 30298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:43.209800 30298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:43.209868 30298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:43.209899 30298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:43.211527 30298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:43.211783 30298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:43.212080 30298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:43.217434 30304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:43.217525 30298 server_base.cc:1034] running on GCE node
W20260926 05:51:43.217555 30303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:43.217432 30306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:43.218199 30298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:43.218482 30298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:43.219647 30298 hybrid_clock.cc:648] HybridClock initialized: now 1790401903219633 us; error 36 us; skew 500 ppm
I20260926 05:51:43.221141 30298 webserver.cc:460] Webserver started at http://127.24.255.1:34399/ using document root <none> and password file <none>
I20260926 05:51:43.221431 30298 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:43.221513 30298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:43.221647 30298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:43.222770 30298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "dea222cb34a049249ed18005f605e5aa"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "981e79691fbd3a6d66737476582cb190"
server_key_iv: "0ebcdccfacf34156f011e26e9c33445c"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.223098 30298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "dea222cb34a049249ed18005f605e5aa"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "981e79691fbd3a6d66737476582cb190"
server_key_iv: "0ebcdccfacf34156f011e26e9c33445c"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.224378 30298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:43.225104 30312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:43.225284 30298 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:51:43.225389 30298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "dea222cb34a049249ed18005f605e5aa"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "981e79691fbd3a6d66737476582cb190"
server_key_iv: "0ebcdccfacf34156f011e26e9c33445c"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.225463 30298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:43.230381 30298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:43.230713 30298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:43.230851 30298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:43.231081 30298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:43.231433 30298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:43.231475 30298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:43.231503 30298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:43.231523 30298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:43.237629 30298 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:35823
I20260926 05:51:43.237697 30425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:35823 every 8 connection(s)
I20260926 05:51:43.238019 30298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:51:43.241523 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.238790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:57067 (local address 127.24.255.62:43879)
0926 05:51:43.238901 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:43.238903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:43.239182 (+ 279us) server_negotiation.cc:407] Connection header received
0926 05:51:43.239724 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:43.239725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:43.239732 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:43.239781 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:43.241234 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:43.241236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:43.241237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:43.241241 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:43.241259 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:43.241263 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:43.241310 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:43.241400 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 05:51:43.241416 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 05:51:43.241966 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.238768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:43879 (local address 127.24.255.1:57067)
0926 05:51:43.238981 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:43.239018 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:43.239612 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:43.239865 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:43.239870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:43.239881 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:43.241061 (+ 1180us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:51:43.241061 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:43.241092 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:51:43.241124 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:43.241325 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:43.241331 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:43.241334 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:51:43.241408 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 05:51:43.241543 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260926 05:51:43.242570 30426 heartbeater.cc:347] Connected to a master server at 127.24.255.62:43879
I20260926 05:51:43.242671 30426 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:43.242835 30426 heartbeater.cc:511] Master 127.24.255.62:43879 requested a full tablet report, sending...
I20260926 05:51:43.243391 30238 ts_manager.cc:194] Registered new tserver with Master: dea222cb34a049249ed18005f605e5aa (127.24.255.1:35823)
I20260926 05:51:43.247140 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 30298
I20260926 05:51:43.247246 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:51:43.247483 25596 external_mini_cluster.cc:1428] Setting key b2345343359710474c595e5c72069bba
I20260926 05:51:43.249050 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:43879
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 05:51:43.335701 30429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:43.335929 30429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:43.335952 30429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:43.337181 30429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:43.337232 30429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:43.337258 30429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:43.338712 30429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:43.338956 30429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:43.339255 30429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:43.344245 30437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:43.344237 30434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:43.344237 30435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:43.344471 30429 server_base.cc:1034] running on GCE node
I20260926 05:51:43.344695 30429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:43.344910 30429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:43.346122 30429 hybrid_clock.cc:648] HybridClock initialized: now 1790401903346115 us; error 115 us; skew 500 ppm
I20260926 05:51:43.347450 30429 webserver.cc:460] Webserver started at http://127.24.255.2:44327/ using document root <none> and password file <none>
I20260926 05:51:43.347679 30429 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:43.347731 30429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:43.347865 30429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:43.349009 30429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "222514eeb8df486e82dd3ee6cfb00ba3"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "74ecda924b66aa88991e73d23322966c"
server_key_iv: "edd6e6f41d22528bd6f01b00c66e748d"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.349357 30429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "222514eeb8df486e82dd3ee6cfb00ba3"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "74ecda924b66aa88991e73d23322966c"
server_key_iv: "edd6e6f41d22528bd6f01b00c66e748d"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.350847 30429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:43.351786 30443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:43.352207 30429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:43.352381 30429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "222514eeb8df486e82dd3ee6cfb00ba3"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "74ecda924b66aa88991e73d23322966c"
server_key_iv: "edd6e6f41d22528bd6f01b00c66e748d"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.352463 30429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:43.356158 30429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:43.356451 30429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:43.356586 30429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:43.356793 30429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:43.357102 30429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:43.357138 30429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:43.357170 30429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:43.357197 30429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:43.363547 30429 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:41341
I20260926 05:51:43.363602 30556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:41341 every 8 connection(s)
I20260926 05:51:43.363946 30429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:51:43.364359 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 30429
I20260926 05:51:43.364461 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:51:43.364643 25596 external_mini_cluster.cc:1428] Setting key 5ec6f0b8614c80a2b33459f81908bc46
I20260926 05:51:43.365862 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:43879
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 05:51:43.367913 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.364732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:35797 (local address 127.24.255.62:43879)
0926 05:51:43.364806 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:43.364809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:43.365056 (+ 247us) server_negotiation.cc:407] Connection header received
0926 05:51:43.365913 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:43.365914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:43.365920 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:43.366008 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:43.367654 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:43.367656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:43.367656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:43.367662 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:43.367682 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:43.367687 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:43.367754 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:43.367808 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 05:51:43.367824 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:51:43.368070 30559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.364699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:43879 (local address 127.24.255.2:35797)
0926 05:51:43.364905 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:43.364954 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:43.365679 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:43.366192 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:43.366198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:43.366208 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:43.367493 (+ 1285us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:51:43.367493 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:43.367527 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:51:43.367558 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:43.367754 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:43.367757 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:43.367759 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:51:43.367811 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 05:51:43.367864 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260926 05:51:43.368633 30557 heartbeater.cc:347] Connected to a master server at 127.24.255.62:43879
I20260926 05:51:43.368739 30557 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:43.368898 30557 heartbeater.cc:511] Master 127.24.255.62:43879 requested a full tablet report, sending...
I20260926 05:51:43.369304 30238 ts_manager.cc:194] Registered new tserver with Master: 222514eeb8df486e82dd3ee6cfb00ba3 (127.24.255.2:41341)
W20260926 05:51:43.449169 30560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:43.449415 30560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:43.449443 30560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:43.450708 30560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:43.450752 30560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:43.450779 30560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:43.452252 30560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:43.452487 30560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:43.452796 30560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:43.458206 30566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:43.458316 30568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:43.458194 30565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:43.458422 30560 server_base.cc:1034] running on GCE node
I20260926 05:51:43.458657 30560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:43.459290 30560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:43.460620 30560 hybrid_clock.cc:648] HybridClock initialized: now 1790401903460591 us; error 41 us; skew 500 ppm
I20260926 05:51:43.462512 30560 webserver.cc:460] Webserver started at http://127.24.255.3:39673/ using document root <none> and password file <none>
I20260926 05:51:43.462777 30560 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:43.462829 30560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:43.462950 30560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:43.464073 30560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "3fe06d6b396e46beb1810b7dd8e0726c"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "7bd90bc1bce8df98926437083e9d4807"
server_key_iv: "fb159ed3ca8d9c3d7e9a07a2e08e513c"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.464586 30560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "3fe06d6b396e46beb1810b7dd8e0726c"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "7bd90bc1bce8df98926437083e9d4807"
server_key_iv: "fb159ed3ca8d9c3d7e9a07a2e08e513c"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.466265 30560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 05:51:43.467257 30574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:43.467571 30560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:43.467671 30560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "3fe06d6b396e46beb1810b7dd8e0726c"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "7bd90bc1bce8df98926437083e9d4807"
server_key_iv: "fb159ed3ca8d9c3d7e9a07a2e08e513c"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.467762 30560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:43.471745 30560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:43.472046 30560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:43.472183 30560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:43.472391 30560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:43.472719 30560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:43.472764 30560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:43.472797 30560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:43.472811 30560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:43.480149 30560 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:33219
I20260926 05:51:43.480189 30687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:33219 every 8 connection(s)
I20260926 05:51:43.480515 30560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:51:43.481531 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 30560
I20260926 05:51:43.481621 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:51:43.481783 25596 external_mini_cluster.cc:1428] Setting key 51f321eb96c2f5b2b84e1d2214b7622d
I20260926 05:51:43.483501 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.481245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:55919 (local address 127.24.255.62:43879)
0926 05:51:43.481320 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:43.481322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:43.481543 (+ 221us) server_negotiation.cc:407] Connection header received
0926 05:51:43.482070 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:43.482071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:43.482078 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:43.482161 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:43.483272 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:43.483274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:43.483274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:43.483279 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:43.483294 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:43.483298 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:43.483346 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:43.483417 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 05:51:43.483432 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:51:43.483742 30690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.481228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:43879 (local address 127.24.255.3:55919)
0926 05:51:43.481406 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:43.481450 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:43.481976 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:43.482237 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:43.482241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:43.482252 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:43.483105 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:51:43.483106 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:43.483146 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:51:43.483183 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:43.483352 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:43.483357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:43.483360 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:51:43.483489 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 05:51:43.483557 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260926 05:51:43.484283 30688 heartbeater.cc:347] Connected to a master server at 127.24.255.62:43879
I20260926 05:51:43.484387 30688 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:43.484563 30688 heartbeater.cc:511] Master 127.24.255.62:43879 requested a full tablet report, sending...
I20260926 05:51:43.484969 30238 ts_manager.cc:194] Registered new tserver with Master: 3fe06d6b396e46beb1810b7dd8e0726c (127.24.255.3:33219)
I20260926 05:51:43.493850 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:51:43.496605 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.495328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.24.255.62:43879)
0926 05:51:43.495415 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:43.495416 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:43.495465 (+ 49us) server_negotiation.cc:407] Connection header received
0926 05:51:43.495488 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:43.495488 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:43.495494 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:43.495538 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:43.496378 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:43.496379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:43.496379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:43.496383 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:43.496399 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:43.496403 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:43.496468 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:43.496523 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 05:51:43.496535 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 05:51:43.498656 30238 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:51:43.500941 30238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:43.506598 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.504429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.24.255.2:41341)
0926 05:51:43.504654 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:43.504658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:43.504673 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:51:43.504821 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:43.504839 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:43.504861 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:43.504934 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:43.506248 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:43.506260 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:43.506261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:43.506267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:43.506306 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:43.506340 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:43.506399 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:43.506479 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 05:51:43.506499 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":112,"threads_started":1}
I20260926 05:51:43.506588 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.504318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:33219 (local address 127.0.0.1:53408)
0926 05:51:43.504736 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:43.504748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:43.504815 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:43.505169 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:43.505175 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:43.505184 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:43.505881 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:51:43.505883 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:43.505909 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:51:43.505946 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:43.506315 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:43.506327 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:43.506332 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:43.506427 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 05:51:43.506447 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":200,"threads_started":1}
I20260926 05:51:43.506785 30697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.504528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:35823 (local address 127.0.0.1:53120)
0926 05:51:43.505017 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:43.505026 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:43.505072 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:43.505565 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:43.505566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:43.505571 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:43.506074 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:51:43.506075 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:43.506081 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:51:43.506090 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:43.506560 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:43.506561 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:43.506562 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:51:43.506600 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 05:51:43.506612 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":67,"threads_started":1}
I20260926 05:51:43.506878 30701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.504773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.24.255.3:33219)
0926 05:51:43.505010 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:43.505014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:43.505025 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:51:43.505054 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:43.505057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:43.505067 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:43.505460 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:43.506062 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:43.506076 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:43.506076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:43.506083 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:43.506162 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:43.506213 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:43.506316 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:43.506766 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 05:51:43.506791 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":129,"threads_started":1}
I20260926 05:51:43.507251 30700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.504735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.24.255.1:35823)
0926 05:51:43.504957 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:43.504960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:43.505123 (+ 163us) server_negotiation.cc:407] Connection header received
0926 05:51:43.505147 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:43.505149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:43.505157 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:43.505219 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:43.506331 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:43.506361 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:43.506361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:43.506369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:43.506424 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:43.506475 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:43.506534 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:43.507136 (+ 602us) server_negotiation.cc:299] Negotiation successful
0926 05:51:43.507162 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":83,"threads_started":1}
I20260926 05:51:43.506587 30696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.504266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:41341 (local address 127.0.0.1:36154)
0926 05:51:43.504499 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:43.504572 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:43.504691 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:43.505395 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:43.505396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:43.505401 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:43.505980 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:51:43.505981 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:43.505991 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:51:43.506010 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:43.506393 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:43.506394 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:43.506395 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:51:43.506443 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:51:43.506459 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":95,"threads_started":1}
I20260926 05:51:43.508111 30491 tablet_service.cc:1467] Processing CreateTablet for tablet 5b44c43df86545729c6322c3341630a7 (DEFAULT_TABLE table=test-table [id=17b73ed3e1ca441985355daecb9741e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:43.508140 30622 tablet_service.cc:1467] Processing CreateTablet for tablet 5b44c43df86545729c6322c3341630a7 (DEFAULT_TABLE table=test-table [id=17b73ed3e1ca441985355daecb9741e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:43.508481 30491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b44c43df86545729c6322c3341630a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:43.508481 30622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b44c43df86545729c6322c3341630a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:43.511256 30703 tablet_bootstrap.cc:492] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c: Bootstrap starting.
I20260926 05:51:43.511257 30702 tablet_bootstrap.cc:492] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3: Bootstrap starting.
I20260926 05:51:43.511945 30702 tablet_bootstrap.cc:654] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:43.511809 30360 tablet_service.cc:1467] Processing CreateTablet for tablet 5b44c43df86545729c6322c3341630a7 (DEFAULT_TABLE table=test-table [id=17b73ed3e1ca441985355daecb9741e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:43.511970 30703 tablet_bootstrap.cc:654] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:43.512180 30360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b44c43df86545729c6322c3341630a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:43.512256 30703 log.cc:824] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:43.512266 30702 log.cc:824] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:43.513044 30702 tablet_bootstrap.cc:492] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3: No bootstrap required, opened a new log
I20260926 05:51:43.513113 30702 ts_tablet_manager.cc:1397] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:51:43.513230 30703 tablet_bootstrap.cc:492] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c: No bootstrap required, opened a new log
I20260926 05:51:43.513301 30703 ts_tablet_manager.cc:1397] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:51:43.515087 30702 raft_consensus.cc:348] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "222514eeb8df486e82dd3ee6cfb00ba3" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41341 } } peers { permanent_uuid: "dea222cb34a049249ed18005f605e5aa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 35823 } } peers { permanent_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33219 } }
I20260926 05:51:43.515232 30702 raft_consensus.cc:374] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:43.515262 30702 raft_consensus.cc:729] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222514eeb8df486e82dd3ee6cfb00ba3, State: Initialized, Role: FOLLOWER
I20260926 05:51:43.515406 30702 consensus_queue.cc:260] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "222514eeb8df486e82dd3ee6cfb00ba3" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41341 } } peers { permanent_uuid: "dea222cb34a049249ed18005f605e5aa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 35823 } } peers { permanent_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33219 } }
I20260926 05:51:43.515679 30702 ts_tablet_manager.cc:1428] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:51:43.515820 30557 heartbeater.cc:503] Master 127.24.255.62:43879 was elected leader, sending a full tablet report...
I20260926 05:51:43.515793 30703 raft_consensus.cc:348] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "222514eeb8df486e82dd3ee6cfb00ba3" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41341 } } peers { permanent_uuid: "dea222cb34a049249ed18005f605e5aa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 35823 } } peers { permanent_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33219 } }
I20260926 05:51:43.515969 30703 raft_consensus.cc:374] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:43.516013 30703 raft_consensus.cc:729] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe06d6b396e46beb1810b7dd8e0726c, State: Initialized, Role: FOLLOWER
I20260926 05:51:43.516119 30703 consensus_queue.cc:260] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "222514eeb8df486e82dd3ee6cfb00ba3" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41341 } } peers { permanent_uuid: "dea222cb34a049249ed18005f605e5aa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 35823 } } peers { permanent_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33219 } }
I20260926 05:51:43.516364 30703 ts_tablet_manager.cc:1428] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:51:43.516443 30688 heartbeater.cc:503] Master 127.24.255.62:43879 was elected leader, sending a full tablet report...
I20260926 05:51:43.516568 30706 tablet_bootstrap.cc:492] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa: Bootstrap starting.
I20260926 05:51:43.517228 30706 tablet_bootstrap.cc:654] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:43.517467 30706 log.cc:824] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:43.518252 30706 tablet_bootstrap.cc:492] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa: No bootstrap required, opened a new log
I20260926 05:51:43.518319 30706 ts_tablet_manager.cc:1397] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:51:43.521004 30706 raft_consensus.cc:348] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "222514eeb8df486e82dd3ee6cfb00ba3" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41341 } } peers { permanent_uuid: "dea222cb34a049249ed18005f605e5aa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 35823 } } peers { permanent_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33219 } }
I20260926 05:51:43.521171 30706 raft_consensus.cc:374] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:43.521198 30706 raft_consensus.cc:729] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea222cb34a049249ed18005f605e5aa, State: Initialized, Role: FOLLOWER
I20260926 05:51:43.521317 30706 consensus_queue.cc:260] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "222514eeb8df486e82dd3ee6cfb00ba3" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41341 } } peers { permanent_uuid: "dea222cb34a049249ed18005f605e5aa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 35823 } } peers { permanent_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33219 } }
I20260926 05:51:43.521605 30706 ts_tablet_manager.cc:1428] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:51:43.521646 30426 heartbeater.cc:503] Master 127.24.255.62:43879 was elected leader, sending a full tablet report...
I20260926 05:51:43.580735 30708 raft_consensus.cc:482] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:43.580878 30708 raft_consensus.cc:504] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "222514eeb8df486e82dd3ee6cfb00ba3" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41341 } } peers { permanent_uuid: "dea222cb34a049249ed18005f605e5aa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 35823 } } peers { permanent_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33219 } }
I20260926 05:51:43.581188 30708 leader_election.cc:288] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 222514eeb8df486e82dd3ee6cfb00ba3 (127.24.255.2:41341), dea222cb34a049249ed18005f605e5aa (127.24.255.1:35823)
I20260926 05:51:43.582827 30690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.581524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:41341 (local address 127.24.255.3:56369)
0926 05:51:43.581582 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:43.581594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:43.581634 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:43.581804 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:43.581805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:43.581808 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:43.582483 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:51:43.582484 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:43.582493 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:51:43.582508 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:43.582711 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:43.582712 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:43.582712 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 05:51:43.582746 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 05:51:43.582760 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:51:43.582908 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.581552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:56369 (local address 127.24.255.2:41341)
0926 05:51:43.581624 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:43.581625 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:43.581635 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:43.581679 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:43.581680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:43.581686 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:43.581743 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:43.582622 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:43.582624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:43.582624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:43.582629 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:43.582649 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:43.582654 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:43.582714 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:43.582749 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 05:51:43.582764 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 05:51:43.582981 30700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.581792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:42003 (local address 127.24.255.1:35823)
0926 05:51:43.581838 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:43.581840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:43.582125 (+ 285us) server_negotiation.cc:407] Connection header received
0926 05:51:43.582183 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:43.582184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:43.582189 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:43.582238 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:43.582684 (+ 446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:43.582685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:43.582686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:43.582689 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:43.582706 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:43.582710 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:43.582762 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:43.582881 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 05:51:43.582896 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:51:43.583140 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.581645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:35823 (local address 127.24.255.3:42003)
0926 05:51:43.581802 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:43.581810 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:43.582122 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:43.582256 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:43.582257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:43.582260 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:43.582590 (+ 330us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:51:43.582591 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:43.582598 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:51:43.582609 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:43.582777 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:43.582778 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:43.582778 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 05:51:43.582864 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 05:51:43.582954 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260926 05:51:43.583348 30511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b44c43df86545729c6322c3341630a7" candidate_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "222514eeb8df486e82dd3ee6cfb00ba3" is_pre_election: true
I20260926 05:51:43.583498 30380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b44c43df86545729c6322c3341630a7" candidate_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea222cb34a049249ed18005f605e5aa" is_pre_election: true
I20260926 05:51:43.583602 30511 raft_consensus.cc:2454] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fe06d6b396e46beb1810b7dd8e0726c in term 0.
I20260926 05:51:43.583639 30380 raft_consensus.cc:2454] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fe06d6b396e46beb1810b7dd8e0726c in term 0.
I20260926 05:51:43.583858 30577 leader_election.cc:302] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe06d6b396e46beb1810b7dd8e0726c, dea222cb34a049249ed18005f605e5aa; no voters:
I20260926 05:51:43.584105 30708 raft_consensus.cc:2781] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:43.584178 30708 raft_consensus.cc:482] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:43.584228 30708 raft_consensus.cc:3037] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:43.584964 30708 raft_consensus.cc:504] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "222514eeb8df486e82dd3ee6cfb00ba3" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41341 } } peers { permanent_uuid: "dea222cb34a049249ed18005f605e5aa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 35823 } } peers { permanent_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33219 } }
I20260926 05:51:43.585083 30708 leader_election.cc:288] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [CANDIDATE]: Term 1 election: Requested vote from peers 222514eeb8df486e82dd3ee6cfb00ba3 (127.24.255.2:41341), dea222cb34a049249ed18005f605e5aa (127.24.255.1:35823)
I20260926 05:51:43.585295 30511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b44c43df86545729c6322c3341630a7" candidate_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "222514eeb8df486e82dd3ee6cfb00ba3"
I20260926 05:51:43.585285 30380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b44c43df86545729c6322c3341630a7" candidate_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea222cb34a049249ed18005f605e5aa"
I20260926 05:51:43.585381 30511 raft_consensus.cc:3037] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:43.585381 30380 raft_consensus.cc:3037] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:43.586122 30380 raft_consensus.cc:2454] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fe06d6b396e46beb1810b7dd8e0726c in term 1.
I20260926 05:51:43.586128 30511 raft_consensus.cc:2454] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fe06d6b396e46beb1810b7dd8e0726c in term 1.
I20260926 05:51:43.586285 30577 leader_election.cc:302] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 222514eeb8df486e82dd3ee6cfb00ba3, 3fe06d6b396e46beb1810b7dd8e0726c; no voters:
I20260926 05:51:43.586372 30708 raft_consensus.cc:2781] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:43.586531 30708 raft_consensus.cc:686] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [term 1 LEADER]: Becoming Leader. State: Replica: 3fe06d6b396e46beb1810b7dd8e0726c, State: Running, Role: LEADER
I20260926 05:51:43.586619 30708 consensus_queue.cc:237] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "222514eeb8df486e82dd3ee6cfb00ba3" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41341 } } peers { permanent_uuid: "dea222cb34a049249ed18005f605e5aa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 35823 } } peers { permanent_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33219 } }
I20260926 05:51:43.587522 30238 catalog_manager.cc:5495] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fe06d6b396e46beb1810b7dd8e0726c (127.24.255.3). New cstate: current_term: 1 leader_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "222514eeb8df486e82dd3ee6cfb00ba3" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dea222cb34a049249ed18005f605e5aa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fe06d6b396e46beb1810b7dd8e0726c" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33219 } health_report { overall_health: HEALTHY } } }
I20260926 05:51:43.603747 30701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.602637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.24.255.3:33219)
0926 05:51:43.602728 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:43.602730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:43.602739 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:43.602773 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:43.602774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:43.602786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:43.602855 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:43.603417 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:43.603418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:43.603418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:43.603444 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:43.603459 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:43.603463 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:43.603513 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:43.603651 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 05:51:43.603667 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 05:51:43.605641 30511 raft_consensus.cc:1261] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3 [term 1 FOLLOWER]: Refusing update from remote peer 3fe06d6b396e46beb1810b7dd8e0726c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:43.605635 30380 raft_consensus.cc:1261] T 5b44c43df86545729c6322c3341630a7 P dea222cb34a049249ed18005f605e5aa [term 1 FOLLOWER]: Refusing update from remote peer 3fe06d6b396e46beb1810b7dd8e0726c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:43.605984 30708 consensus_queue.cc:1035] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dea222cb34a049249ed18005f605e5aa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:43.606145 30712 consensus_queue.cc:1035] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [LEADER]: Connected to new peer: Peer: permanent_uuid: "222514eeb8df486e82dd3ee6cfb00ba3" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:43.609061 30716 mvcc.cc:205] Tried to move back new op lower bound from 7333486197166088192 to 7333486197091459072. Current Snapshot: MvccSnapshot[applied={T|T < 7333486197166088192}]
I20260926 05:51:43.609324 30717 mvcc.cc:205] Tried to move back new op lower bound from 7333486197166088192 to 7333486197091459072. Current Snapshot: MvccSnapshot[applied={T|T < 7333486197166088192}]
I20260926 05:51:43.611513 30237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:43.611606 30237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:43.612406 30237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1a7b67045ae4638b987c0320839906b: Sending DeleteTablet for 3 replicas of tablet 5b44c43df86545729c6322c3341630a7
I20260926 05:51:43.612721 30622 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b44c43df86545729c6322c3341630a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:43 UTC) from {username='slave'} at 127.0.0.1:53408
I20260926 05:51:43.612829 30491 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b44c43df86545729c6322c3341630a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:43 UTC) from {username='slave'} at 127.0.0.1:36154
I20260926 05:51:43.612982 30724 tablet_replica.cc:330] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c: stopping tablet replica
I20260926 05:51:43.613073 30724 raft_consensus.cc:2229] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:51:43.613080 30360 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b44c43df86545729c6322c3341630a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:43 UTC) from {username='slave'} at 127.0.0.1:53120
I20260926 05:51:43.613132 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 30298
I20260926 05:51:43.613686 30724 raft_consensus.cc:2258] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:43.614526 30725 tablet_replica.cc:330] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3: stopping tablet replica
I20260926 05:51:43.614733 30725 raft_consensus.cc:2229] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:43.614845 30725 raft_consensus.cc:2258] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:43.615249 30725 ts_tablet_manager.cc:1905] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:43.616755 30725 ts_tablet_manager.cc:1918] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:43.616823 30725 log.cc:1196] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/5b44c43df86545729c6322c3341630a7
I20260926 05:51:43.617029 30725 ts_tablet_manager.cc:1939] T 5b44c43df86545729c6322c3341630a7 P 222514eeb8df486e82dd3ee6cfb00ba3: Deleting consensus metadata
I20260926 05:51:43.617725 30724 ts_tablet_manager.cc:1905] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:43.618280 30224 catalog_manager.cc:4841] TS 222514eeb8df486e82dd3ee6cfb00ba3 (127.24.255.2:41341): tablet 5b44c43df86545729c6322c3341630a7 (table test-table [id=17b73ed3e1ca441985355daecb9741e6]) successfully deleted
W20260926 05:51:43.618434 30224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:35823 (error 108)
I20260926 05:51:43.618722 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 30429
I20260926 05:51:43.618896 30697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.618757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:35823 (local address 127.0.0.1:53128)
0926 05:51:43.618813 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:43.618858 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:35823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:51:43.619920 30724 ts_tablet_manager.cc:1918] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:43.619995 30724 log.cc:1196] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/5b44c43df86545729c6322c3341630a7
I20260926 05:51:43.620242 30724 ts_tablet_manager.cc:1939] T 5b44c43df86545729c6322c3341630a7 P 3fe06d6b396e46beb1810b7dd8e0726c: Deleting consensus metadata
I20260926 05:51:43.620687 30223 catalog_manager.cc:4841] TS 3fe06d6b396e46beb1810b7dd8e0726c (127.24.255.3:33219): tablet 5b44c43df86545729c6322c3341630a7 (table test-table [id=17b73ed3e1ca441985355daecb9741e6]) successfully deleted
W20260926 05:51:43.621284 30224 catalog_manager.cc:4584] TS dea222cb34a049249ed18005f605e5aa (127.24.255.1:35823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b44c43df86545729c6322c3341630a7: Network error: Client connection negotiation failed: client connection to 127.24.255.1:35823: connect: Connection refused (error 111)
I20260926 05:51:43.624368 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 30560
I20260926 05:51:43.630890 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 30207
2026-09-26T05:51:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (696 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 05:51:43.650620 25596 mini_oidc.cc:152] Starting JWKS server
I20260926 05:51:43.651742 25596 webserver.cc:460] Webserver started at https://127.0.0.1:38545/ using document root <none> and password file <none>
I20260926 05:51:43.651860 25596 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 05:51:43.652038 25596 webserver.cc:460] Webserver started at http://127.0.0.1:45079/ using document root <none> and password file <none>
2026-09-26T05:51:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:43Z Disabled control of system clock
I20260926 05:51:43.659732 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:36797
--webserver_interface=127.24.255.62
--webserver_port=0
--builtin_ntp_servers=127.24.255.20:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:36797
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38545/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/testchainca.cert with env {}
W20260926 05:51:43.743374 30732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:43.743642 30732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:43.743680 30732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:43.745132 30732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:43.745190 30732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:43.745206 30732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:43.745223 30732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:43.745239 30732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 05:51:43.745254 30732 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 05:51:43.745270 30732 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38545/jwks.json
I20260926 05:51:43.747138 30732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:36797
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:36797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38545/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:43.747435 30732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:43.747646 30732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:43.753700 30738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:43.753691 30740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:43.753691 30737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:43.753994 30732 server_base.cc:1034] running on GCE node
I20260926 05:51:43.754390 30732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:43.754751 30732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:43.755940 30732 hybrid_clock.cc:648] HybridClock initialized: now 1790401903755921 us; error 35 us; skew 500 ppm
I20260926 05:51:43.757370 30732 webserver.cc:460] Webserver started at http://127.24.255.62:34201/ using document root <none> and password file <none>
I20260926 05:51:43.757619 30732 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:43.757671 30732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:43.757807 30732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:43.759126 30732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "bd89961d943b4b6a84a5fdfaca4a328b"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "2528467f62a8d584f5fbdb6379581c3d"
server_key_iv: "c2c9866ca59564a1f36c770f9b0aa678"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.759476 30732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "bd89961d943b4b6a84a5fdfaca4a328b"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "2528467f62a8d584f5fbdb6379581c3d"
server_key_iv: "c2c9866ca59564a1f36c770f9b0aa678"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.760881 30732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:51:43.761700 30746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:43.761965 30732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:43.762068 30732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "bd89961d943b4b6a84a5fdfaca4a328b"
format_stamp: "Formatted at 2026-09-26 05:51:43 on dist-test-slave-7fr8"
server_key: "2528467f62a8d584f5fbdb6379581c3d"
server_key_iv: "c2c9866ca59564a1f36c770f9b0aa678"
server_key_version: "encryptionkey@0"
I20260926 05:51:43.762172 30732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:43.782331 30732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:43.782689 30732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:43.782836 30732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:43.786815 30732 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:36797
I20260926 05:51:43.786888 30798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:36797 every 8 connection(s)
I20260926 05:51:43.787211 30732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:43.788437 30799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:43.790885 30799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b: Bootstrap starting.
I20260926 05:51:43.791438 30799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:43.791667 30799 log.cc:824] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:43.792379 30799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b: No bootstrap required, opened a new log
I20260926 05:51:43.794056 30799 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd89961d943b4b6a84a5fdfaca4a328b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 36797 } }
I20260926 05:51:43.794269 30799 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:43.794297 30799 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd89961d943b4b6a84a5fdfaca4a328b, State: Initialized, Role: FOLLOWER
I20260926 05:51:43.794425 30799 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd89961d943b4b6a84a5fdfaca4a328b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 36797 } }
I20260926 05:51:43.794525 30799 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:43.794550 30799 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:43.794579 30799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:43.794814 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 30732
I20260926 05:51:43.794910 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:43.795077 25596 external_mini_cluster.cc:1428] Setting key 0f026c554882ffaedfd1f14953723617
I20260926 05:51:43.795145 30799 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd89961d943b4b6a84a5fdfaca4a328b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 36797 } }
I20260926 05:51:43.795250 30799 leader_election.cc:302] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd89961d943b4b6a84a5fdfaca4a328b; no voters:
I20260926 05:51:43.795435 30799 leader_election.cc:288] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:43.795563 30802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:43.795719 30802 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [term 1 LEADER]: Becoming Leader. State: Replica: bd89961d943b4b6a84a5fdfaca4a328b, State: Running, Role: LEADER
I20260926 05:51:43.795727 30799 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:43.795836 30802 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd89961d943b4b6a84a5fdfaca4a328b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 36797 } }
I20260926 05:51:43.796296 30805 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd89961d943b4b6a84a5fdfaca4a328b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd89961d943b4b6a84a5fdfaca4a328b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 36797 } } }
I20260926 05:51:43.796376 30805 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:43.796383 30804 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd89961d943b4b6a84a5fdfaca4a328b. Latest consensus state: current_term: 1 leader_uuid: "bd89961d943b4b6a84a5fdfaca4a328b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd89961d943b4b6a84a5fdfaca4a328b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 36797 } } }
I20260926 05:51:43.796571 30804 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:43.796671 30813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:43.797508 30813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:43.799892 30813 catalog_manager.cc:1347] Generated new cluster ID: 26bf8789b2ad46f5a55ff0ce02028353
I20260926 05:51:43.800020 30813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:43.802467 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.795746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35900 (local address 127.24.255.62:36797)
0926 05:51:43.795974 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:43.795994 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:43.796602 (+ 608us) server_negotiation.cc:407] Connection header received
0926 05:51:43.796839 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:43.796849 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:43.796935 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:43.797284 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:43.797929 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:43.799216 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:43.800422 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:43.800979 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:43.801098 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:43.801747 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:43.801752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:43.801752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:43.801933 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:43.801976 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:43.801989 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:43.802045 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:43.802159 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 05:51:43.802247 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":65,"threads_started":1}
I20260926 05:51:43.803503 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:43.834899 30813 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:43.835654 30813 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:43.852334 30813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd89961d943b4b6a84a5fdfaca4a328b: Generated new TSK 0
I20260926 05:51:43.852572 30813 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 05:51:43.876827 30829 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 05:51:43.876946 30829 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 05:51:43.877190 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:43.861695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.24.255.62:36797)
0926 05:51:43.861744 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:43.861746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:43.861899 (+ 153us) server_negotiation.cc:407] Connection header received
0926 05:51:43.861983 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:43.861984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:43.861997 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:43.862061 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0926 05:51:43.862282 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:43.862773 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:43.863923 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:43.864116 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:43.864179 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:43.864445 (+ 266us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 05:51:43.876953 (+ 12508us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 05:51:43.877122 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35904: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 05:51:43.877244 30806 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35904: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 05:51:43.877579 30828 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:43.861587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:36797 (local address 127.0.0.1:35904)
0926 05:51:43.861780 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:43.861792 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:43.861879 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:43.862087 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:43.862088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:43.862090 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 05:51:43.862194 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:43.862197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:43.862877 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:43.862878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:43.863795 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:43.863798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:43.864199 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:43.864200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:43.864276 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:43.864307 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 05:51:43.877132 (+ 12825us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 05:51:43.877244 (+ 112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.255.62:36797: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.24.255.62:36797: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 05:51:43.878165 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 30732
2026-09-26T05:51:43Z chronyd exiting
I20260926 05:51:43.914172 25596 test_util.cc:175] -----------------------------------------------
I20260926 05:51:43.914254 25596 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401881171543-25596-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (264 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 05:51:43.915028 25596 mini_oidc.cc:152] Starting JWKS server
I20260926 05:51:43.915726 25596 webserver.cc:460] Webserver started at https://127.0.0.1:38483/ using document root <none> and password file <none>
I20260926 05:51:43.915771 25596 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 05:51:43.915906 25596 webserver.cc:460] Webserver started at http://127.0.0.1:40865/ using document root <none> and password file <none>
2026-09-26T05:51:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:43Z Disabled control of system clock
I20260926 05:51:43.924432 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:45637
--webserver_interface=127.24.255.62
--webserver_port=0
--builtin_ntp_servers=127.24.255.20:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:45637
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38483/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/testchainca.cert with env {}
W20260926 05:51:44.015413 30835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:44.015682 30835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:44.015707 30835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:44.017130 30835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:44.017198 30835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:44.017211 30835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:44.017225 30835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:44.017235 30835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 05:51:44.017247 30835 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 05:51:44.017257 30835 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38483/jwks.json
I20260926 05:51:44.019573 30835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:45637
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:45637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38483/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:44.019896 30835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:44.020134 30835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:51:44.025300 30835 server_base.cc:1034] running on GCE node
W20260926 05:51:44.025264 30840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:44.025405 30843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:44.025264 30841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:44.025810 30835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:44.026114 30835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:44.027312 30835 hybrid_clock.cc:648] HybridClock initialized: now 1790401904027276 us; error 54 us; skew 500 ppm
I20260926 05:51:44.029035 30835 webserver.cc:460] Webserver started at http://127.24.255.62:41645/ using document root <none> and password file <none>
I20260926 05:51:44.029268 30835 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:44.029333 30835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:44.029431 30835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:44.030614 30835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "f9b3bf3600c242b4ac611d6755b920bb"
format_stamp: "Formatted at 2026-09-26 05:51:44 on dist-test-slave-7fr8"
server_key: "6751fa02fc4ceb3d52c12ee513104c96"
server_key_iv: "e69a8ae162fd436abdf06a5f16965dc1"
server_key_version: "encryptionkey@0"
I20260926 05:51:44.030951 30835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "f9b3bf3600c242b4ac611d6755b920bb"
format_stamp: "Formatted at 2026-09-26 05:51:44 on dist-test-slave-7fr8"
server_key: "6751fa02fc4ceb3d52c12ee513104c96"
server_key_iv: "e69a8ae162fd436abdf06a5f16965dc1"
server_key_version: "encryptionkey@0"
I20260926 05:51:44.032352 30835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:44.033313 30849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:44.033758 30835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:44.033878 30835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "f9b3bf3600c242b4ac611d6755b920bb"
format_stamp: "Formatted at 2026-09-26 05:51:44 on dist-test-slave-7fr8"
server_key: "6751fa02fc4ceb3d52c12ee513104c96"
server_key_iv: "e69a8ae162fd436abdf06a5f16965dc1"
server_key_version: "encryptionkey@0"
I20260926 05:51:44.033972 30835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:44.045089 30835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:44.045416 30835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:44.045536 30835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:44.049968 30835 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:45637
I20260926 05:51:44.050004 30901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:45637 every 8 connection(s)
I20260926 05:51:44.050359 30835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:44.050586 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 30835
I20260926 05:51:44.050699 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:44.050882 25596 external_mini_cluster.cc:1428] Setting key 4d7bd028d666c11778eb04cf393a66bc
I20260926 05:51:44.051131 30902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:44.054734 30902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb: Bootstrap starting.
I20260926 05:51:44.055476 30902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:44.055750 30902 log.cc:824] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:44.056527 30902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb: No bootstrap required, opened a new log
I20260926 05:51:44.056908 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:44.051376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.24.255.62:45637)
0926 05:51:44.051589 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:44.051596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:44.051621 (+ 25us) server_negotiation.cc:407] Connection header received
0926 05:51:44.052070 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:44.052076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:44.052139 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:44.052255 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:44.052515 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.053379 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.054528 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.055037 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.055140 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:44.055886 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:44.055892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:44.055892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:44.056062 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:44.056097 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:44.056110 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:44.056163 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:44.056250 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 05:51:44.056339 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260926 05:51:44.058394 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:44.059008 30902 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b3bf3600c242b4ac611d6755b920bb" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45637 } }
I20260926 05:51:44.059209 30902 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:44.059266 30902 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b3bf3600c242b4ac611d6755b920bb, State: Initialized, Role: FOLLOWER
I20260926 05:51:44.059432 30902 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b3bf3600c242b4ac611d6755b920bb" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45637 } }
I20260926 05:51:44.059523 30902 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:44.059556 30902 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:44.059594 30902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:44.060245 30902 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b3bf3600c242b4ac611d6755b920bb" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45637 } }
I20260926 05:51:44.060364 30902 leader_election.cc:302] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b3bf3600c242b4ac611d6755b920bb; no voters:
I20260926 05:51:44.060532 30902 leader_election.cc:288] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:44.060583 30907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:44.060743 30907 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [term 1 LEADER]: Becoming Leader. State: Replica: f9b3bf3600c242b4ac611d6755b920bb, State: Running, Role: LEADER
I20260926 05:51:44.060810 30902 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:44.060839 30907 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b3bf3600c242b4ac611d6755b920bb" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45637 } }
I20260926 05:51:44.061249 30909 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b3bf3600c242b4ac611d6755b920bb. Latest consensus state: current_term: 1 leader_uuid: "f9b3bf3600c242b4ac611d6755b920bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b3bf3600c242b4ac611d6755b920bb" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45637 } } }
I20260926 05:51:44.061241 30908 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9b3bf3600c242b4ac611d6755b920bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b3bf3600c242b4ac611d6755b920bb" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45637 } } }
I20260926 05:51:44.061354 30908 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:44.061342 30909 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:44.061560 30915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:44.062048 30915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:44.063663 30915 catalog_manager.cc:1347] Generated new cluster ID: f4efbaf08aea41fe9044b927d127dd86
I20260926 05:51:44.063717 30915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:44.074168 30915 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:44.074918 30915 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:44.089027 30915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9b3bf3600c242b4ac611d6755b920bb: Generated new TSK 0
I20260926 05:51:44.089263 30915 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 05:51:44.114161 30932 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 05:51:44.114257 30932 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 05:51:44.114603 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:44.099908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.24.255.62:45637)
0926 05:51:44.099989 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:44.099991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:44.100007 (+ 16us) server_negotiation.cc:407] Connection header received
0926 05:51:44.100104 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:44.100106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:44.100120 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:44.100185 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0926 05:51:44.100322 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.100838 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.101618 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.101813 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.101870 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:44.102062 (+ 192us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 05:51:44.114367 (+ 12305us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38483/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 05:51:44.114531 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55980: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38483/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 05:51:44.114663 30905 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55980: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38483/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 05:51:44.114789 30931 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:44.099789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:45637 (local address 127.0.0.1:55980)
0926 05:51:44.099949 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:44.099963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:44.100025 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:44.100194 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:44.100195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:44.100197 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 05:51:44.100263 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:44.100265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.100928 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.100929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:44.101534 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:44.101536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.101900 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.101901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:44.101954 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:44.101959 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 05:51:44.114583 (+ 12624us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38483/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 05:51:44.114698 (+ 115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.255.62:45637: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38483/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260926 05:51:44.115145 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 30835
2026-09-26T05:51:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 05:51:44.152945 25596 mini_oidc.cc:152] Starting JWKS server
I20260926 05:51:44.153630 25596 webserver.cc:460] Webserver started at https://127.0.0.1:34673/ using document root <none> and password file <none>
I20260926 05:51:44.153672 25596 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 05:51:44.153784 25596 webserver.cc:460] Webserver started at http://127.0.0.1:38369/ using document root <none> and password file <none>
2026-09-26T05:51:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:44Z Disabled control of system clock
I20260926 05:51:44.161720 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:44673
--webserver_interface=127.24.255.62
--webserver_port=0
--builtin_ntp_servers=127.24.255.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:44673
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34673/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 05:51:44.250240 30938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:44.250469 30938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:44.250494 30938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:44.251742 30938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:44.251789 30938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:44.251801 30938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:44.251811 30938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:44.251822 30938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 05:51:44.251833 30938 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 05:51:44.251843 30938 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34673/jwks.json
I20260926 05:51:44.253286 30938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:44673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:44673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34673/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:44.253535 30938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:44.253808 30938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:44.259320 30946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:44.259410 30944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:44.259450 30943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:44.260078 30938 server_base.cc:1034] running on GCE node
I20260926 05:51:44.260480 30938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:44.260985 30938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:44.262200 30938 hybrid_clock.cc:648] HybridClock initialized: now 1790401904262136 us; error 93 us; skew 500 ppm
I20260926 05:51:44.263650 30938 webserver.cc:460] Webserver started at http://127.24.255.62:32853/ using document root <none> and password file <none>
I20260926 05:51:44.263901 30938 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:44.263955 30938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:44.264089 30938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:44.265179 30938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "1528bb215abd416db278babfba42da7a"
format_stamp: "Formatted at 2026-09-26 05:51:44 on dist-test-slave-7fr8"
server_key: "f829dab0325ce07ba01a580cc8a41d69"
server_key_iv: "c2ecc3ee6ec78661cfdc881819a19d2c"
server_key_version: "encryptionkey@0"
I20260926 05:51:44.265528 30938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "1528bb215abd416db278babfba42da7a"
format_stamp: "Formatted at 2026-09-26 05:51:44 on dist-test-slave-7fr8"
server_key: "f829dab0325ce07ba01a580cc8a41d69"
server_key_iv: "c2ecc3ee6ec78661cfdc881819a19d2c"
server_key_version: "encryptionkey@0"
I20260926 05:51:44.267195 30938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:44.268126 30952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:44.268317 30938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:44.268416 30938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "1528bb215abd416db278babfba42da7a"
format_stamp: "Formatted at 2026-09-26 05:51:44 on dist-test-slave-7fr8"
server_key: "f829dab0325ce07ba01a580cc8a41d69"
server_key_iv: "c2ecc3ee6ec78661cfdc881819a19d2c"
server_key_version: "encryptionkey@0"
I20260926 05:51:44.268499 30938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:44.282275 30938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:44.282617 30938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:44.282747 30938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:44.287114 31004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:44673 every 8 connection(s)
I20260926 05:51:44.287144 30938 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:44673
I20260926 05:51:44.287618 30938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:44.288373 31005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:44.291296 31005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a: Bootstrap starting.
I20260926 05:51:44.291865 31005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:44.292240 31005 log.cc:824] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:44.293015 31005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a: No bootstrap required, opened a new log
I20260926 05:51:44.294729 31005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1528bb215abd416db278babfba42da7a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44673 } }
I20260926 05:51:44.294889 31005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:44.294912 31005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1528bb215abd416db278babfba42da7a, State: Initialized, Role: FOLLOWER
I20260926 05:51:44.295035 31005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1528bb215abd416db278babfba42da7a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44673 } }
I20260926 05:51:44.295112 31005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:44.295136 31005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:44.295173 31005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:44.295737 31005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1528bb215abd416db278babfba42da7a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44673 } }
I20260926 05:51:44.295835 31005 leader_election.cc:302] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1528bb215abd416db278babfba42da7a; no voters:
I20260926 05:51:44.295969 31005 leader_election.cc:288] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:44.296103 31008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:44.296203 31005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:44.296260 31008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [term 1 LEADER]: Becoming Leader. State: Replica: 1528bb215abd416db278babfba42da7a, State: Running, Role: LEADER
I20260926 05:51:44.296495 31008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1528bb215abd416db278babfba42da7a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44673 } }
I20260926 05:51:44.296912 31009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1528bb215abd416db278babfba42da7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1528bb215abd416db278babfba42da7a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44673 } } }
I20260926 05:51:44.297000 31009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:44.297086 31010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1528bb215abd416db278babfba42da7a. Latest consensus state: current_term: 1 leader_uuid: "1528bb215abd416db278babfba42da7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1528bb215abd416db278babfba42da7a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44673 } } }
I20260926 05:51:44.297158 31010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:44.297205 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 30938
I20260926 05:51:44.297302 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:44.297448 31017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:44.297504 25596 external_mini_cluster.cc:1428] Setting key d203f09a1876ca518a307226e28e3743
I20260926 05:51:44.298061 31017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:44.299635 31017 catalog_manager.cc:1347] Generated new cluster ID: 496639934ec84c2eb70820d0041d2b97
I20260926 05:51:44.299691 31017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:44.304293 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:44.297984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.24.255.62:44673)
0926 05:51:44.298715 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:44.298723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:44.298749 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:51:44.298840 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:44.298846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:44.298901 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:44.299041 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:44.299265 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.300619 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.301770 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.302490 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.302632 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:44.303474 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:44.303478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:44.303479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:44.303713 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:44.303769 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:44.303786 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:44.303863 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:44.303964 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 05:51:44.304060 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":78,"threads_started":1}
I20260926 05:51:44.305473 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:44.310307 31017 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:44.310801 31017 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:44.315474 31017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1528bb215abd416db278babfba42da7a: Generated new TSK 0
I20260926 05:51:44.315696 31017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:44.326879 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:44.322811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.24.255.62:44673)
0926 05:51:44.322862 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:44.322864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:44.322925 (+ 61us) server_negotiation.cc:407] Connection header received
0926 05:51:44.323018 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:44.323019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:44.323032 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:44.323101 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0926 05:51:44.323280 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.324086 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.325248 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.325476 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.325542 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:44.325725 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 05:51:44.326658 (+ 933us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 05:51:44.326791 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 05:51:44.326963 31034 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:44.322664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44673 (local address 127.0.0.1:39668)
0926 05:51:44.322848 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:44.322859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:44.322932 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:44.323113 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:44.323114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:44.323116 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 05:51:44.323186 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:44.323188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.324196 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.324196 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:44.325145 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:44.325148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.325560 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.325561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:44.325628 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:44.325634 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 05:51:44.326764 (+ 1130us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 05:51:44.326863 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.255.62:44673: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":120,"threads_started":1}
I20260926 05:51:44.327260 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 30938
E20260926 05:51:44.337913 31035 webserver.cc:533] Webserver: sslize failed:
E20260926 05:51:44.338019 31035 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T05:51:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T05:51:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:44Z Disabled control of system clock
I20260926 05:51:44.372809 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:34219
--webserver_interface=127.24.255.62
--webserver_port=0
--builtin_ntp_servers=127.24.255.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:34219
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/testchainca.cert with env {}
W20260926 05:51:44.458638 31039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:44.458871 31039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:44.458895 31039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:44.460374 31039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:44.460446 31039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:44.460459 31039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:44.460480 31039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:44.460492 31039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 05:51:44.460505 31039 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 05:51:44.460515 31039 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 05:51:44.462354 31039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:34219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:34219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:44.462620 31039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:44.462841 31039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:44.467731 31047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:44.467914 31045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:44.468008 31044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:44.468008 31039 server_base.cc:1034] running on GCE node
I20260926 05:51:44.468318 31039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:44.468637 31039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:44.469833 31039 hybrid_clock.cc:648] HybridClock initialized: now 1790401904469804 us; error 46 us; skew 500 ppm
I20260926 05:51:44.471284 31039 webserver.cc:460] Webserver started at http://127.24.255.62:44493/ using document root <none> and password file <none>
I20260926 05:51:44.471560 31039 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:44.471616 31039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:44.471737 31039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:44.472796 31039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "a2a7e4263402489c975481f5cea8992c"
format_stamp: "Formatted at 2026-09-26 05:51:44 on dist-test-slave-7fr8"
server_key: "ccea06a9192f02acb50510faa5adf6ad"
server_key_iv: "47cfae16069e53040388b01b2ae67e3a"
server_key_version: "encryptionkey@0"
I20260926 05:51:44.473209 31039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "a2a7e4263402489c975481f5cea8992c"
format_stamp: "Formatted at 2026-09-26 05:51:44 on dist-test-slave-7fr8"
server_key: "ccea06a9192f02acb50510faa5adf6ad"
server_key_iv: "47cfae16069e53040388b01b2ae67e3a"
server_key_version: "encryptionkey@0"
I20260926 05:51:44.474566 31039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:51:44.475557 31053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:44.476181 31039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:44.476361 31039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "a2a7e4263402489c975481f5cea8992c"
format_stamp: "Formatted at 2026-09-26 05:51:44 on dist-test-slave-7fr8"
server_key: "ccea06a9192f02acb50510faa5adf6ad"
server_key_iv: "47cfae16069e53040388b01b2ae67e3a"
server_key_version: "encryptionkey@0"
I20260926 05:51:44.476465 31039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:44.486697 31039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:44.487046 31039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:44.487186 31039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:44.490870 31039 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:34219
I20260926 05:51:44.490909 31105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:34219 every 8 connection(s)
I20260926 05:51:44.491241 31039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:44.492215 31106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:44.494627 31106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c: Bootstrap starting.
I20260926 05:51:44.495172 31106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:44.495409 31106 log.cc:824] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:44.496070 31106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c: No bootstrap required, opened a new log
I20260926 05:51:44.497499 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 31039
I20260926 05:51:44.497605 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:44.497774 25596 external_mini_cluster.cc:1428] Setting key e6c02c83330528869f2f3ad08f87dc87
I20260926 05:51:44.497854 31106 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7e4263402489c975481f5cea8992c" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34219 } }
I20260926 05:51:44.498020 31106 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:44.498044 31106 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a7e4263402489c975481f5cea8992c, State: Initialized, Role: FOLLOWER
I20260926 05:51:44.498421 31106 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7e4263402489c975481f5cea8992c" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34219 } }
I20260926 05:51:44.498528 31106 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:44.498562 31106 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:44.498612 31106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:44.499279 31106 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7e4263402489c975481f5cea8992c" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34219 } }
I20260926 05:51:44.499367 31106 leader_election.cc:302] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a7e4263402489c975481f5cea8992c; no voters:
I20260926 05:51:44.499503 31106 leader_election.cc:288] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:44.499559 31111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:44.499701 31111 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [term 1 LEADER]: Becoming Leader. State: Replica: a2a7e4263402489c975481f5cea8992c, State: Running, Role: LEADER
I20260926 05:51:44.499754 31106 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:44.499805 31111 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7e4263402489c975481f5cea8992c" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34219 } }
I20260926 05:51:44.500156 31111 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2a7e4263402489c975481f5cea8992c. Latest consensus state: current_term: 1 leader_uuid: "a2a7e4263402489c975481f5cea8992c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7e4263402489c975481f5cea8992c" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34219 } } }
I20260926 05:51:44.500231 31111 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:44.500360 31112 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2a7e4263402489c975481f5cea8992c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7e4263402489c975481f5cea8992c" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34219 } } }
I20260926 05:51:44.500414 31112 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:44.500506 31118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:44.502166 31118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:44.503810 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:44.498263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.24.255.62:34219)
0926 05:51:44.498484 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:44.498492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:44.498526 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:51:44.498655 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:44.498663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:44.498732 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:44.498851 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:44.499094 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.500799 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.501617 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.502265 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.502394 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:44.503062 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:44.503065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:44.503066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:44.503275 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:44.503352 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:44.503368 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:44.503433 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:44.503513 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 05:51:44.503597 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260926 05:51:44.504132 31118 catalog_manager.cc:1347] Generated new cluster ID: 61463fa3dc524a8284fc03334a8c5d76
I20260926 05:51:44.504187 31118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:44.504707 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:44.515156 31118 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:44.515697 31118 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:44.517560 31118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2a7e4263402489c975481f5cea8992c: Generated new TSK 0
I20260926 05:51:44.517752 31118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:44.993736 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:44.528893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.24.255.62:34219)
0926 05:51:44.528978 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:44.528980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:44.528995 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:51:44.529167 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:44.529169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:44.529185 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:44.529271 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0926 05:51:44.529514 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.530084 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.531374 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.531640 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.531748 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:44.532031 (+ 283us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 05:51:44.993475 (+461444us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 05:51:44.993671 (+ 196us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39106: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 05:51:44.993810 31110 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39106: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 05:51:44.993873 31135 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:44.528720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34219 (local address 127.0.0.1:39106)
0926 05:51:44.528929 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:44.528941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:44.528998 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:44.529293 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:44.529293 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:44.529296 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 05:51:44.529379 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:44.529382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.530215 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.530217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:44.531188 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:44.531191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:44.531770 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:44.531771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:44.531848 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:44.531857 (+ 9us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 05:51:44.993635 (+461778us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 05:51:44.993767 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.255.62:34219: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":120,"threads_started":1}
I20260926 05:51:44.994267 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 31039
2026-09-26T05:51:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (646 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (93 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:45 dist-test-slave-7fr8 krb5kdc[31147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:45 dist-test-slave-7fr8 krb5kdc[31147](info): set up 2 sockets
Sep 26 05:51:45 dist-test-slave-7fr8 krb5kdc[31147](info): commencing operation
krb5kdc: starting...
W20260926 05:51:46.146368 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:46 dist-test-slave-7fr8 krb5kdc[31147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:51:46.270336 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:45707
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:45707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:46.352903 31160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:46.353135 31160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:46.353159 31160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:46.354395 31160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:46.354436 31160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:46.354449 31160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:46.354461 31160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:46.354478 31160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:46.356045 31160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:45707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:45707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:46.356273 31160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:46.356477 31160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:46.361606 31168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:46.361706 31160 server_base.cc:1034] running on GCE node
W20260926 05:51:46.361730 31165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:46.361752 31166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:46.362176 31160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:46.362502 31160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:46.363705 31160 hybrid_clock.cc:648] HybridClock initialized: now 1790401906363659 us; error 73 us; skew 500 ppm
Sep 26 05:51:46 dist-test-slave-7fr8 krb5kdc[31147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401906, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:46.375029 31160 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:51:46.375535 31160 webserver.cc:460] Webserver started at http://127.24.255.62:44177/ using document root <none> and password file <none>
I20260926 05:51:46.375754 31160 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:46.375800 31160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:46.375902 31160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:46.377084 31160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "77b87642b0734428b9327d166c161715"
format_stamp: "Formatted at 2026-09-26 05:51:46 on dist-test-slave-7fr8"
server_key: "eb008147ad61a5ffdf4c4dbcdadc3537"
server_key_iv: "1de56a83a2d542a682069c221c8d194b"
server_key_version: "encryptionkey@0"
I20260926 05:51:46.377420 31160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "77b87642b0734428b9327d166c161715"
format_stamp: "Formatted at 2026-09-26 05:51:46 on dist-test-slave-7fr8"
server_key: "eb008147ad61a5ffdf4c4dbcdadc3537"
server_key_iv: "1de56a83a2d542a682069c221c8d194b"
server_key_version: "encryptionkey@0"
I20260926 05:51:46.378914 31160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:51:46.379832 31175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:46.380111 31160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:46.380223 31160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "77b87642b0734428b9327d166c161715"
format_stamp: "Formatted at 2026-09-26 05:51:46 on dist-test-slave-7fr8"
server_key: "eb008147ad61a5ffdf4c4dbcdadc3537"
server_key_iv: "1de56a83a2d542a682069c221c8d194b"
server_key_version: "encryptionkey@0"
I20260926 05:51:46.380311 31160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:46.429888 31160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:46.432457 31160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:46.432622 31160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:46.436467 31160 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:45707
I20260926 05:51:46.436571 31227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:45707 every 8 connection(s)
I20260926 05:51:46.436879 31160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:46.437860 31228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:46.440227 31228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715: Bootstrap starting.
I20260926 05:51:46.440798 31228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:46.441092 31228 log.cc:824] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:46.441829 31228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715: No bootstrap required, opened a new log
I20260926 05:51:46.443547 31228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b87642b0734428b9327d166c161715" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45707 } }
I20260926 05:51:46.443702 31228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:46.443723 31228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b87642b0734428b9327d166c161715, State: Initialized, Role: FOLLOWER
I20260926 05:51:46.443951 31228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b87642b0734428b9327d166c161715" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45707 } }
I20260926 05:51:46.444067 31228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:46.444095 31228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:46.444146 31228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:46.444748 31228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b87642b0734428b9327d166c161715" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45707 } }
I20260926 05:51:46.444840 31228 leader_election.cc:302] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77b87642b0734428b9327d166c161715; no voters:
I20260926 05:51:46.445009 31228 leader_election.cc:288] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:46.445050 31231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:46.445274 31228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:46.445369 31231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [term 1 LEADER]: Becoming Leader. State: Replica: 77b87642b0734428b9327d166c161715, State: Running, Role: LEADER
I20260926 05:51:46.445514 31231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b87642b0734428b9327d166c161715" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45707 } }
I20260926 05:51:46.445866 31232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77b87642b0734428b9327d166c161715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b87642b0734428b9327d166c161715" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45707 } } }
I20260926 05:51:46.445966 31232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:46.445886 31233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77b87642b0734428b9327d166c161715. Latest consensus state: current_term: 1 leader_uuid: "77b87642b0734428b9327d166c161715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77b87642b0734428b9327d166c161715" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45707 } } }
I20260926 05:51:46.446007 31233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:46.446033 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 31160
I20260926 05:51:46.446146 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:46.446328 25596 external_mini_cluster.cc:1428] Setting key c12aab6d874b8fd5f5666796f0f61f1d
W20260926 05:51:46.447352 31247 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:51:46.447412 31247 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:46.447464 31248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:46.447603 31248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:46.449322 31248 catalog_manager.cc:1347] Generated new cluster ID: c088dad1dccf4cd8b961710afc565f3d
I20260926 05:51:46.449381 31248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:51:46 dist-test-slave-7fr8 krb5kdc[31147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:46.456672 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:46.447064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.24.255.62:45707)
0926 05:51:46.447236 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:46.447246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:46.447281 (+ 35us) server_negotiation.cc:407] Connection header received
0926 05:51:46.447412 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:46.447419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:46.447489 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:46.447634 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:46.448597 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.449830 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.450680 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.451113 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.451211 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:46.453486 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:46.453491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:46.453492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:46.453505 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:46.455140 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:46.455494 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:46.455497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:46.455497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:46.455592 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:46.455776 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:46.455778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:46.455778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:46.456056 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:46.456214 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:46.456298 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 05:51:46.456446 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260926 05:51:46.464901 31248 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:46.465811 31248 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:46.470278 31248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77b87642b0734428b9327d166c161715: Generated new TSK 0
I20260926 05:51:46.470507 31248 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:51:46.505985 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:45707
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:46.591728 31254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:46.591970 31254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:46.592005 31254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:46.593297 31254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:46.593350 31254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:46.593379 31254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:46.595091 31254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:46.595358 31254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:46.595592 31254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:46.601022 31259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:46.601056 31260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:46.601035 31262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:46.601254 31254 server_base.cc:1034] running on GCE node
I20260926 05:51:46.601543 31254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:46.601851 31254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:46.603010 31254 hybrid_clock.cc:648] HybridClock initialized: now 1790401906602999 us; error 34 us; skew 500 ppm
Sep 26 05:51:46 dist-test-slave-7fr8 krb5kdc[31147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401906, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:46.606159 31254 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:51:46.606657 31254 webserver.cc:460] Webserver started at http://127.24.255.1:36529/ using document root <none> and password file <none>
I20260926 05:51:46.606873 31254 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:46.606921 31254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:46.607054 31254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:46.608162 31254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "2500a70390454515bc1704dd5db3d61e"
format_stamp: "Formatted at 2026-09-26 05:51:46 on dist-test-slave-7fr8"
server_key: "deb42a67d184518ffcb3b2c4dfb4ef94"
server_key_iv: "099a0701def1bbcbf47bf63a0b3b013f"
server_key_version: "encryptionkey@0"
I20260926 05:51:46.608515 31254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "2500a70390454515bc1704dd5db3d61e"
format_stamp: "Formatted at 2026-09-26 05:51:46 on dist-test-slave-7fr8"
server_key: "deb42a67d184518ffcb3b2c4dfb4ef94"
server_key_iv: "099a0701def1bbcbf47bf63a0b3b013f"
server_key_version: "encryptionkey@0"
I20260926 05:51:46.610000 31254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:46.611049 31269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:46.611248 31254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:46.611349 31254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "2500a70390454515bc1704dd5db3d61e"
format_stamp: "Formatted at 2026-09-26 05:51:46 on dist-test-slave-7fr8"
server_key: "deb42a67d184518ffcb3b2c4dfb4ef94"
server_key_iv: "099a0701def1bbcbf47bf63a0b3b013f"
server_key_version: "encryptionkey@0"
I20260926 05:51:46.611425 31254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:46.620795 31254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:46.623158 31254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:46.623287 31254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:46.623504 31254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:46.623822 31254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:46.623857 31254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:46.623881 31254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:46.623893 31254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:46.631168 31254 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:39665
I20260926 05:51:46.631232 31382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:39665 every 8 connection(s)
I20260926 05:51:46.631609 31254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:51:46 dist-test-slave-7fr8 krb5kdc[31147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401906, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:46.641636 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 31254
I20260926 05:51:46.641742 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:51:46.641939 25596 external_mini_cluster.cc:1428] Setting key f49e004dfbae7ba5d69998eef59ec5be
I20260926 05:51:46.643203 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:46.632834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:33569 (local address 127.24.255.62:45707)
0926 05:51:46.632903 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:46.632905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:46.633168 (+ 263us) server_negotiation.cc:407] Connection header received
0926 05:51:46.634302 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:46.634304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:46.634330 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:46.634396 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:46.635154 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.635667 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.637158 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.637391 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.637457 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:46.639983 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:46.639985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:46.639985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:46.639993 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:46.641430 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:46.642164 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:46.642165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:46.642165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:46.642238 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:46.642502 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:46.642504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:46.642504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:46.642622 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:46.642687 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:46.643012 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 05:51:46.643111 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:51:46.643322 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:46.632772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:45707 (local address 127.24.255.1:33569)
0926 05:51:46.632989 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:46.633036 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:46.634003 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:46.634519 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:46.634524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:46.634825 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:46.635038 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:46.635043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.635870 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.635872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:46.637036 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:46.637043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.637510 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.637511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:46.637639 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:46.638329 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:46.638339 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:46.639869 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:46.641560 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:46.641575 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:46.641578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:46.641989 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:46.642309 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:46.642310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:46.642311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:46.642429 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:46.642692 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:46.642699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:46.642845 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 05:51:46.642964 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 05:51:46.643111 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":98,"threads_started":1}
I20260926 05:51:46.644254 31383 heartbeater.cc:347] Connected to a master server at 127.24.255.62:45707
I20260926 05:51:46.644354 31383 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:46.644541 31383 heartbeater.cc:511] Master 127.24.255.62:45707 requested a full tablet report, sending...
I20260926 05:51:46.645081 31192 ts_manager.cc:194] Registered new tserver with Master: 2500a70390454515bc1704dd5db3d61e (127.24.255.1:39665)
I20260926 05:51:46.645742 31192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:33569
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:51:46.670663 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:45707
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:46.752636 31388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:46.752877 31388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:46.752902 31388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:46.754424 31388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:46.754496 31388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:46.754527 31388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:46.756512 31388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:46.756788 31388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:46.757066 31388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:46.762884 31396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:46.763013 31388 server_base.cc:1034] running on GCE node
W20260926 05:51:46.762871 31394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:46.762871 31393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:46.763275 31388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:46.763545 31388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:46.764734 31388 hybrid_clock.cc:648] HybridClock initialized: now 1790401906764706 us; error 47 us; skew 500 ppm
Sep 26 05:51:46 dist-test-slave-7fr8 krb5kdc[31147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401906, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:46.767611 31388 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:51:46.768151 31388 webserver.cc:460] Webserver started at http://127.24.255.2:43707/ using document root <none> and password file <none>
I20260926 05:51:46.768361 31388 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:46.768412 31388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:46.768551 31388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:46.769629 31388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "ccf1568d57c7453cb6c38e1969fe4a7f"
format_stamp: "Formatted at 2026-09-26 05:51:46 on dist-test-slave-7fr8"
server_key: "83015875c378339d9d38a8ac538b7fc1"
server_key_iv: "000035e4181c636d8d1ed5c4b817bb1e"
server_key_version: "encryptionkey@0"
I20260926 05:51:46.769979 31388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "ccf1568d57c7453cb6c38e1969fe4a7f"
format_stamp: "Formatted at 2026-09-26 05:51:46 on dist-test-slave-7fr8"
server_key: "83015875c378339d9d38a8ac538b7fc1"
server_key_iv: "000035e4181c636d8d1ed5c4b817bb1e"
server_key_version: "encryptionkey@0"
I20260926 05:51:46.771452 31388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:46.772311 31403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:46.772535 31388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:46.772632 31388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "ccf1568d57c7453cb6c38e1969fe4a7f"
format_stamp: "Formatted at 2026-09-26 05:51:46 on dist-test-slave-7fr8"
server_key: "83015875c378339d9d38a8ac538b7fc1"
server_key_iv: "000035e4181c636d8d1ed5c4b817bb1e"
server_key_version: "encryptionkey@0"
I20260926 05:51:46.772707 31388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:46.785583 31388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:46.787896 31388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:46.788066 31388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:46.788297 31388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:46.788611 31388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:46.788645 31388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:46.788676 31388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:46.788691 31388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:46.794947 31388 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:37703
I20260926 05:51:46.795043 31516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:37703 every 8 connection(s)
I20260926 05:51:46.795411 31388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:51:46.795959 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 31388
I20260926 05:51:46.796046 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:51:46.796237 25596 external_mini_cluster.cc:1428] Setting key a92b725fe95219b7b712828679a155eb
Sep 26 05:51:46 dist-test-slave-7fr8 krb5kdc[31147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401906, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
I20260926 05:51:46.805296 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:46.796227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:44253 (local address 127.24.255.62:45707)
0926 05:51:46.796304 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:46.796306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:46.796563 (+ 257us) server_negotiation.cc:407] Connection header received
0926 05:51:46.797552 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:46.797553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:46.797566 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:46.797612 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:46.798288 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.798796 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.799784 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.799992 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.800048 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:46.802395 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:46.802398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:46.802398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:46.802407 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:46.803902 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:46.804327 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:46.804328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:46.804329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:46.804391 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:46.804640 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:46.804641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:46.804642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:46.804769 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:46.804825 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:46.805109 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 05:51:46.805203 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:51:46.805473 31519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:46.796202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:45707 (local address 127.24.255.2:44253)
0926 05:51:46.796358 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:46.796404 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:46.797439 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:46.797688 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:46.797691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:46.797967 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:46.798204 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:46.798209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.798897 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.798899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:46.799693 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:46.799695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.800071 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.800072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:46.800190 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:46.800686 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:46.800695 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:46.802227 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:46.803993 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:46.804007 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:46.804009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:46.804250 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:46.804453 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:46.804454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:46.804455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:46.804564 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:46.804829 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:46.804836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:46.804985 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 05:51:46.805078 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 05:51:46.805247 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260926 05:51:46.805989 31517 heartbeater.cc:347] Connected to a master server at 127.24.255.62:45707
I20260926 05:51:46.806123 31517 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
I20260926 05:51:46.806317 31517 heartbeater.cc:511] Master 127.24.255.62:45707 requested a full tablet report, sending...
I20260926 05:51:46.806734 31192 ts_manager.cc:194] Registered new tserver with Master: ccf1568d57c7453cb6c38e1969fe4a7f (127.24.255.2:37703)
I20260926 05:51:46.807145 31192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:44253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:51:46.821681 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:45707
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:46.904887 31522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:46.905123 31522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:46.905149 31522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:46.906404 31522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:46.906452 31522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:46.906481 31522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:46.908063 31522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:46.908295 31522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:46.908486 31522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:46.913710 31530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:46.913844 31522 server_base.cc:1034] running on GCE node
W20260926 05:51:46.913743 31528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:46.913722 31527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:46.914191 31522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:46.914470 31522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:46.915683 31522 hybrid_clock.cc:648] HybridClock initialized: now 1790401906915675 us; error 49 us; skew 500 ppm
Sep 26 05:51:46 dist-test-slave-7fr8 krb5kdc[31147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401906, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:46.918325 31522 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:51:46.918756 31522 webserver.cc:460] Webserver started at http://127.24.255.3:41255/ using document root <none> and password file <none>
I20260926 05:51:46.918957 31522 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:46.919004 31522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:46.919099 31522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:46.920080 31522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "16f2cdb5191d496ab945d6d386cb8725"
format_stamp: "Formatted at 2026-09-26 05:51:46 on dist-test-slave-7fr8"
server_key: "31a8950f55cecac3d31ff74764ab9c42"
server_key_iv: "547336fbd11c831743dd741f4be2508a"
server_key_version: "encryptionkey@0"
I20260926 05:51:46.920374 31522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "16f2cdb5191d496ab945d6d386cb8725"
format_stamp: "Formatted at 2026-09-26 05:51:46 on dist-test-slave-7fr8"
server_key: "31a8950f55cecac3d31ff74764ab9c42"
server_key_iv: "547336fbd11c831743dd741f4be2508a"
server_key_version: "encryptionkey@0"
I20260926 05:51:46.921664 31522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:51:46.922448 31537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:46.922646 31522 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:46.922739 31522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "16f2cdb5191d496ab945d6d386cb8725"
format_stamp: "Formatted at 2026-09-26 05:51:46 on dist-test-slave-7fr8"
server_key: "31a8950f55cecac3d31ff74764ab9c42"
server_key_iv: "547336fbd11c831743dd741f4be2508a"
server_key_version: "encryptionkey@0"
I20260926 05:51:46.922823 31522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:46.943063 31522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:46.945571 31522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:46.945735 31522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:46.945977 31522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:46.946389 31522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:46.946432 31522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:46.946468 31522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:46.946496 31522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:46.953123 31522 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:43905
I20260926 05:51:46.953198 31650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:43905 every 8 connection(s)
I20260926 05:51:46.953531 31522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:51:46.957188 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 31522
I20260926 05:51:46.957381 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:51:46.957595 25596 external_mini_cluster.cc:1428] Setting key 1b82bf257fe4e0e9f935dd6d4e81b668
Sep 26 05:51:46 dist-test-slave-7fr8 krb5kdc[31147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401906, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:46.966421 31653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:46.954335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:45707 (local address 127.24.255.3:51215)
0926 05:51:46.954581 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:46.954707 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:46.956060 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:46.956450 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:46.956453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:46.956812 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:46.957126 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:46.957134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.958207 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.958209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:46.959721 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:46.959725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.960124 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.960125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:46.960264 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:46.960784 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:46.960793 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:46.962605 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:46.964522 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:46.964532 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:46.964534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:46.964785 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:46.965023 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:46.965024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:46.965024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:46.965209 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:46.965557 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:46.965563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:46.965749 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 05:51:46.965867 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 05:51:46.966054 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":99,"threads_started":1}
I20260926 05:51:46.966472 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:46.954480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:51215 (local address 127.24.255.62:45707)
0926 05:51:46.954596 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:46.954598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:46.954834 (+ 236us) server_negotiation.cc:407] Connection header received
0926 05:51:46.956250 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:46.956252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:46.956269 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:46.956321 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:46.957284 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.958088 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.959841 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:46.960045 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:46.960103 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:46.962727 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:46.962729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:46.962730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:46.962740 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:46.964423 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:46.964866 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:46.964867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:46.964868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:46.964940 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:46.965296 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:46.965298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:46.965298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:46.965483 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:46.965557 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:46.966060 (+ 503us) server_negotiation.cc:299] Negotiation successful
0926 05:51:46.966340 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 05:51:46.968120 31651 heartbeater.cc:347] Connected to a master server at 127.24.255.62:45707
I20260926 05:51:46.968245 31651 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:46.968453 31651 heartbeater.cc:511] Master 127.24.255.62:45707 requested a full tablet report, sending...
I20260926 05:51:46.968847 31192 ts_manager.cc:194] Registered new tserver with Master: 16f2cdb5191d496ab945d6d386cb8725 (127.24.255.3:43905)
I20260926 05:51:46.969369 31192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:51215
I20260926 05:51:46.969427 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/test-admin.keytab.
Sep 26 05:51:46 dist-test-slave-7fr8 krb5kdc[31147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:46.995441 25596 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 05:51:46.997709 31658 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:46.996472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.24.255.1:39665)
0926 05:51:46.996640 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:46.996645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:46.996659 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:51:46.996729 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:46.996731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:46.996747 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:46.996814 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:46.997628 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49834: BlockingRecv error: recv got EOF from 127.0.0.1:49834 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
W20260926 05:51:46.997840 25596 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/krb5cc' of type FILE
Sep 26 05:51:46 dist-test-slave-7fr8 krb5kdc[31147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:46.998975 25596 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 05:51:46 dist-test-slave-7fr8 krb5kdc[31147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:47.000558 25596 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 05:51:47 dist-test-slave-7fr8 krb5kdc[31147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.1@KRBTEST.COM
I20260926 05:51:47.011849 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:47.001565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.24.255.1:39665)
0926 05:51:47.001622 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:47.001624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:47.001652 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:51:47.001726 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:47.001726 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:47.001737 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:47.001785 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:47.002364 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:47.003406 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:47.004485 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:47.004962 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:47.005041 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:47.008260 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:47.008275 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:47.008275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:47.008284 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:47.010013 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:47.010811 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:47.010814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:47.010814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:47.010885 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:47.011164 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:47.011165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:47.011166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:47.011356 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:47.011506 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:47.011653 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 05:51:47.011736 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 05:51:47.014353 31658 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:47.013734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.24.255.1:39665)
0926 05:51:47.013790 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:47.013792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:47.013824 (+ 32us) server_negotiation.cc:407] Connection header received
0926 05:51:47.013898 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:47.013899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:47.013910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:47.013958 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:47.014301 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49856: BlockingRecv error: recv got EOF from 127.0.0.1:49856 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 05:51:47.014570 25596 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/krb5cc)
Sep 26 05:51:47 dist-test-slave-7fr8 krb5kdc[31147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:47.015801 25596 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 05:51:47 dist-test-slave-7fr8 krb5kdc[31147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:47.016806 25596 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 05:51:47 dist-test-slave-7fr8 krb5kdc[31147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.1@KRBTEST.COM
I20260926 05:51:47.026069 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:47.017613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.24.255.1:39665)
0926 05:51:47.017716 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:47.017718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:47.017729 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:51:47.017752 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:47.017753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:47.017761 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:47.017815 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:47.018580 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:47.019115 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:47.020040 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:47.020254 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:47.020318 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:47.022857 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:47.022859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:47.022859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:47.022867 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:47.024481 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:47.024976 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:47.024979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:47.024980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:47.025054 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:47.025397 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:47.025398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:47.025399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:47.025521 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:47.025625 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:47.025861 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 05:51:47.025975 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 05:51:47.027897 31658 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:47.027274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.24.255.1:39665)
0926 05:51:47.027317 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:47.027319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:47.027345 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:51:47.027425 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:47.027425 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:47.027434 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:47.027478 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:47.027783 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49886: BlockingRecv error: recv got EOF from 127.0.0.1:49886 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:51:47.028084 25596 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401881171543-25596-0/krb5kdc/krb5cc' of type FILE
Sep 26 05:51:47 dist-test-slave-7fr8 krb5kdc[31147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:47.029297 25596 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 05:51:47 dist-test-slave-7fr8 krb5kdc[31147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:47.030381 25596 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 05:51:47 dist-test-slave-7fr8 krb5kdc[31147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.1@KRBTEST.COM
I20260926 05:51:47.038082 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:47.031064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.24.255.1:39665)
0926 05:51:47.031119 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:47.031121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:47.031131 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:47.031154 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:47.031155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:47.031163 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:47.031213 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:47.031660 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:47.032170 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:47.032949 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:47.033157 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:47.033222 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:47.035481 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:47.035483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:47.035483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:47.035490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:47.036833 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:47.037204 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:47.037205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:47.037205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:47.037265 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:47.037547 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:47.037548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:47.037548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:47.037668 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:47.037756 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:47.037935 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 05:51:47.038003 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:51:47.038800 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 31254
I20260926 05:51:47.043421 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 31388
I20260926 05:51:47.047488 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 31522
I20260926 05:51:47.052455 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 31160
2026-09-26T05:51:47Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1875 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:47 dist-test-slave-7fr8 krb5kdc[31671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:47 dist-test-slave-7fr8 krb5kdc[31671](info): set up 2 sockets
Sep 26 05:51:47 dist-test-slave-7fr8 krb5kdc[31671](info): commencing operation
krb5kdc: starting...
W20260926 05:51:48.042519 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:48 dist-test-slave-7fr8 krb5kdc[31671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.62.keytab.
Entry for principal oryx/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.62.keytab.
I20260926 05:51:48.169279 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:46751
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.62.keytab
--principal=oryx/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:46751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:48.255587 31684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:48.255820 31684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:48.255853 31684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:48.257192 31684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:48.257256 31684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:48.257273 31684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:48.257290 31684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:48.257306 31684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:48.259263 31684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:46751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.62.keytab
--principal=oryx/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:46751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:48.259656 31684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:48.259902 31684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:48.265195 31692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:48.265230 31684 server_base.cc:1034] running on GCE node
W20260926 05:51:48.265200 31689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:48.265200 31690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:48.265673 31684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:48.266047 31684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:48.267248 31684 hybrid_clock.cc:648] HybridClock initialized: now 1790401908267166 us; error 94 us; skew 500 ppm
Sep 26 05:51:48 dist-test-slave-7fr8 krb5kdc[31671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:48.269953 31684 init.cc:377] Logged in from keytab as oryx/127.24.255.62@KRBTEST.COM (short username oryx)
I20260926 05:51:48.270516 31684 webserver.cc:460] Webserver started at http://127.24.255.62:44945/ using document root <none> and password file <none>
I20260926 05:51:48.270751 31684 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:48.270802 31684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:48.270934 31684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:48.272099 31684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "59ebed1574644d97a63c34f3b3107cef"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "92cb87395614f6f31b96164cb4db85d3"
server_key_iv: "eb6b554eb1143bf3bbf0b9be45ade794"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.272455 31684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "59ebed1574644d97a63c34f3b3107cef"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "92cb87395614f6f31b96164cb4db85d3"
server_key_iv: "eb6b554eb1143bf3bbf0b9be45ade794"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.273998 31684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:48.274874 31699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:48.275206 31684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:48.275375 31684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "59ebed1574644d97a63c34f3b3107cef"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "92cb87395614f6f31b96164cb4db85d3"
server_key_iv: "eb6b554eb1143bf3bbf0b9be45ade794"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.275453 31684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:48.286851 31684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:48.289163 31684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:48.289376 31684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:48.293183 31684 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:46751
I20260926 05:51:48.293262 31751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:46751 every 8 connection(s)
I20260926 05:51:48.293586 31684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:48.294337 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 31684
I20260926 05:51:48.294441 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:48.294680 25596 external_mini_cluster.cc:1428] Setting key b8e1ad137c3edcd931bc3c669ef1aff9
I20260926 05:51:48.294814 31752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:48.298441 31752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef: Bootstrap starting.
I20260926 05:51:48.299081 31752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:48.299383 31752 log.cc:824] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:48.300685 31752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef: No bootstrap required, opened a new log
Sep 26 05:51:48 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:48.302917 31752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } }
I20260926 05:51:48.303097 31752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:48.303138 31752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ebed1574644d97a63c34f3b3107cef, State: Initialized, Role: FOLLOWER
I20260926 05:51:48.303284 31752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } }
I20260926 05:51:48.303366 31752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:48.303402 31752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:48.303457 31752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:48.304044 31752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } }
I20260926 05:51:48.304140 31752 leader_election.cc:302] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59ebed1574644d97a63c34f3b3107cef; no voters:
I20260926 05:51:48.304344 31752 leader_election.cc:288] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:48.304404 31757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:48.304570 31757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [term 1 LEADER]: Becoming Leader. State: Replica: 59ebed1574644d97a63c34f3b3107cef, State: Running, Role: LEADER
I20260926 05:51:48.304620 31752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:48.304663 31757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } }
I20260926 05:51:48.305218 31757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59ebed1574644d97a63c34f3b3107cef. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } }
I20260926 05:51:48.305305 31757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:48.305259 31758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } }
I20260926 05:51:48.305526 31758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:48.306226 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:48.295243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57430 (local address 127.24.255.62:46751)
0926 05:51:48.295455 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:48.295472 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:48.295503 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:51:48.295949 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:48.295958 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:48.296022 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:48.296178 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:48.296809 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.298258 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.299112 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.299714 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.299820 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:48.302395 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:48.302400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:48.302400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:48.302413 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:48.304079 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:48.304502 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:48.304505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:48.304506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:48.304621 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:48.305038 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:48.305039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:48.305039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:48.305392 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:48.305585 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:48.305834 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 05:51:48.305983 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":122,"threads_started":1}
W20260926 05:51:48.307161 31772 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:51:48.307219 31772 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:48.307293 31765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:48.307423 31765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:48.309053 31765 catalog_manager.cc:1347] Generated new cluster ID: 76801b8588544dd9add52d55f500263a
I20260926 05:51:48.309110 31765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:48.312700 31765 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:48.313267 31765 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:48.317011 31765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef: Generated new TSK 0
I20260926 05:51:48.317230 31765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab.
Entry for principal oryx/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab.
I20260926 05:51:48.349864 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:46751
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab
--principal=oryx/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:48.454460 31778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:48.454710 31778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:48.454746 31778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:48.456147 31778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:48.456203 31778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:48.456245 31778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:48.458125 31778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab
--principal=oryx/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:46751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:48.458501 31778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:48.458743 31778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:48.464325 31783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:48.464349 31784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:48.464653 31786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:48.464643 31778 server_base.cc:1034] running on GCE node
I20260926 05:51:48.464922 31778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:48.465186 31778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:48.466369 31778 hybrid_clock.cc:648] HybridClock initialized: now 1790401908466319 us; error 71 us; skew 500 ppm
Sep 26 05:51:48 dist-test-slave-7fr8 krb5kdc[31671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:48.469908 31778 init.cc:377] Logged in from keytab as oryx/127.24.255.1@KRBTEST.COM (short username oryx)
I20260926 05:51:48.470486 31778 webserver.cc:460] Webserver started at http://127.24.255.1:43339/ using document root <none> and password file <none>
I20260926 05:51:48.470760 31778 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:48.470822 31778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:48.470984 31778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:48.472458 31778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "e1c39876dc73428d87aadbba9ed9de58"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "04899793c074fee7618634aca44eec04"
server_key_iv: "6a77f9ea74025eb9ae88398261972d94"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.472914 31778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "e1c39876dc73428d87aadbba9ed9de58"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "04899793c074fee7618634aca44eec04"
server_key_iv: "6a77f9ea74025eb9ae88398261972d94"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.474751 31778 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 05:51:48.475579 31793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:48.475925 31778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:48.476027 31778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "e1c39876dc73428d87aadbba9ed9de58"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "04899793c074fee7618634aca44eec04"
server_key_iv: "6a77f9ea74025eb9ae88398261972d94"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.476114 31778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:48.514797 31778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:48.516927 31778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:48.517083 31778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:48.517330 31778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:48.517632 31778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:48.517664 31778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:48.517685 31778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:48.517707 31778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:48.523386 31778 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:39449
I20260926 05:51:48.523474 31906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:39449 every 8 connection(s)
I20260926 05:51:48.523753 31778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:51:48.527055 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 31778
I20260926 05:51:48.527256 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:51:48.527485 25596 external_mini_cluster.cc:1428] Setting key 2ea3bdb9ea5ed4cd4bac1e868e64c62e
Sep 26 05:51:48 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:48.534348 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:48.524542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:37631 (local address 127.24.255.62:46751)
0926 05:51:48.524606 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:48.524608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:48.524830 (+ 222us) server_negotiation.cc:407] Connection header received
0926 05:51:48.525280 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:48.525281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:48.525292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:48.525355 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:48.526441 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.526987 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.528239 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.528452 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.528519 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:48.530998 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:48.530999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:48.530999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:48.531007 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:48.532589 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:48.533121 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:48.533122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:48.533123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:48.533211 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:48.533510 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:48.533511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:48.533511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:48.533677 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:48.533758 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:48.534141 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 05:51:48.534244 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:51:48.534549 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:48.524505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.1:37631)
0926 05:51:48.524679 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:48.524721 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:48.525202 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:48.525420 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:48.525424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:48.526023 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:48.526285 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:48.526291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.527124 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.527125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:48.528135 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:48.528138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.528540 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.528541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:48.528689 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:48.529421 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:48.529431 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:48.530881 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:48.532709 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:48.532718 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:48.532721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:48.533004 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:48.533313 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:48.533314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:48.533314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:48.533436 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:48.533744 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:48.533748 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:48.533882 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 05:51:48.534111 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 05:51:48.534313 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260926 05:51:48.535151 31907 heartbeater.cc:347] Connected to a master server at 127.24.255.62:46751
I20260926 05:51:48.535254 31907 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:48.535446 31907 heartbeater.cc:511] Master 127.24.255.62:46751 requested a full tablet report, sending...
I20260926 05:51:48.536055 31716 ts_manager.cc:194] Registered new tserver with Master: e1c39876dc73428d87aadbba9ed9de58 (127.24.255.1:39449)
WARNING: no policy specified for oryx/127.24.255.2@KRBTEST.COM; defaulting to no policy
I20260926 05:51:48.536845 31716 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:37631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab.
Entry for principal oryx/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab.
I20260926 05:51:48.559062 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:46751
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab
--principal=oryx/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:48.644236 31912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:48.644461 31912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:48.644487 31912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:48.645774 31912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:48.645823 31912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:48.645853 31912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:48.647490 31912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab
--principal=oryx/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:46751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:48.647750 31912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:48.648008 31912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:48.653124 31918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:48.653133 31917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:48.653215 31912 server_base.cc:1034] running on GCE node
W20260926 05:51:48.653136 31920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:48.653555 31912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:48.653838 31912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:48.655009 31912 hybrid_clock.cc:648] HybridClock initialized: now 1790401908654988 us; error 34 us; skew 500 ppm
Sep 26 05:51:48 dist-test-slave-7fr8 krb5kdc[31671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:48.657581 31912 init.cc:377] Logged in from keytab as oryx/127.24.255.2@KRBTEST.COM (short username oryx)
I20260926 05:51:48.658074 31912 webserver.cc:460] Webserver started at http://127.24.255.2:41765/ using document root <none> and password file <none>
I20260926 05:51:48.658318 31912 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:48.658368 31912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:48.658490 31912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:48.659546 31912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "0513b0a4ecb94eb1935b1eda3d179c35"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "1d534e9639b2ebafd99524170b43ffca"
server_key_iv: "f2183880d738b989d81614e22bdec3cd"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.659888 31912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "0513b0a4ecb94eb1935b1eda3d179c35"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "1d534e9639b2ebafd99524170b43ffca"
server_key_iv: "f2183880d738b989d81614e22bdec3cd"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.661306 31912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:51:48.662223 31927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:48.662397 31912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:48.662484 31912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "0513b0a4ecb94eb1935b1eda3d179c35"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "1d534e9639b2ebafd99524170b43ffca"
server_key_iv: "f2183880d738b989d81614e22bdec3cd"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.662564 31912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:48.678742 31912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:48.681789 31912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:48.681944 31912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:48.682209 31912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:48.682539 31912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:48.682582 31912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:48.682612 31912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:48.682632 31912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:48.689127 31912 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:37079
I20260926 05:51:48.689275 32040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:37079 every 8 connection(s)
I20260926 05:51:48.689553 31912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:51:48.694466 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 31912
I20260926 05:51:48.694579 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:51:48.694779 25596 external_mini_cluster.cc:1428] Setting key 377964bc1398c185f3bf0e3d2169d5e0
Sep 26 05:51:48 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:48.700532 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:48.690514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:59057 (local address 127.24.255.62:46751)
0926 05:51:48.690572 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:48.690574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:48.690648 (+ 74us) server_negotiation.cc:407] Connection header received
0926 05:51:48.691178 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:48.691180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:48.691192 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:48.691259 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:48.692350 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.692897 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.694007 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.694303 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.694401 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:48.697128 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:48.697130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:48.697130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:48.697140 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:48.699030 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:48.699571 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:48.699573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:48.699573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:48.699629 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:48.699863 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:48.699865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:48.699865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:48.699985 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:48.700044 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:48.700327 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 05:51:48.700428 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:51:48.700696 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:48.690337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.2:59057)
0926 05:51:48.690539 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:48.690584 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:48.691056 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:48.691318 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:48.691322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:48.691978 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:48.692211 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:48.692216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.693047 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.693048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:48.693881 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:48.693884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.694482 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.694483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:48.694612 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:48.695248 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:48.695259 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:48.697019 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:48.699137 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:48.699148 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:48.699151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:48.699497 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:48.699691 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:48.699692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:48.699692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:48.699801 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:48.700047 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:48.700054 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:48.700201 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 05:51:48.700290 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 05:51:48.700468 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
I20260926 05:51:48.701258 32041 heartbeater.cc:347] Connected to a master server at 127.24.255.62:46751
I20260926 05:51:48.701381 32041 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:48.701587 32041 heartbeater.cc:511] Master 127.24.255.62:46751 requested a full tablet report, sending...
I20260926 05:51:48.702032 31716 ts_manager.cc:194] Registered new tserver with Master: 0513b0a4ecb94eb1935b1eda3d179c35 (127.24.255.2:37079)
I20260926 05:51:48.702427 31716 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:59057
WARNING: no policy specified for oryx/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab.
Entry for principal oryx/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab.
I20260926 05:51:48.722394 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:46751
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab
--principal=oryx/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:48.806257 32046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:48.806473 32046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:48.806499 32046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:48.807814 32046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:48.807873 32046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:48.807901 32046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:48.809518 32046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab
--principal=oryx/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:46751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:48.809751 32046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:48.809955 32046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:48.815255 32054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:48.815378 32051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:48.815255 32052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:48.815605 32046 server_base.cc:1034] running on GCE node
I20260926 05:51:48.815832 32046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:48.816110 32046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:48.817296 32046 hybrid_clock.cc:648] HybridClock initialized: now 1790401908817278 us; error 42 us; skew 500 ppm
Sep 26 05:51:48 dist-test-slave-7fr8 krb5kdc[31671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:48.820232 32046 init.cc:377] Logged in from keytab as oryx/127.24.255.3@KRBTEST.COM (short username oryx)
I20260926 05:51:48.820729 32046 webserver.cc:460] Webserver started at http://127.24.255.3:39813/ using document root <none> and password file <none>
I20260926 05:51:48.820958 32046 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:48.821022 32046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:48.821151 32046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:48.822281 32046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "e0390f7fabb244bc902af3b9eab49ded"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "b30a4aa20fe6528c598580e48183ee62"
server_key_iv: "7f8a142c2ee30d263a62d41dbd916f32"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.822619 32046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "e0390f7fabb244bc902af3b9eab49ded"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "b30a4aa20fe6528c598580e48183ee62"
server_key_iv: "7f8a142c2ee30d263a62d41dbd916f32"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.824000 32046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:48.824874 32061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:48.825079 32046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:48.825167 32046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "e0390f7fabb244bc902af3b9eab49ded"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "b30a4aa20fe6528c598580e48183ee62"
server_key_iv: "7f8a142c2ee30d263a62d41dbd916f32"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.825237 32046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:48.837692 32046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:48.839856 32046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:48.840022 32046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:48.840255 32046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:48.840562 32046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:48.840597 32046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:48.840626 32046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:48.840643 32046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:48.846845 32046 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:46849
I20260926 05:51:48.846906 32174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:46849 every 8 connection(s)
I20260926 05:51:48.847221 32046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:51:48.847623 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 32046
I20260926 05:51:48.847697 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:51:48.847901 25596 external_mini_cluster.cc:1428] Setting key 9920608825cc78a673afaaceaba9c448
Sep 26 05:51:48 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:48.857417 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:48.848089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:47153 (local address 127.24.255.62:46751)
0926 05:51:48.848165 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:48.848167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:48.848358 (+ 191us) server_negotiation.cc:407] Connection header received
0926 05:51:48.848944 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:48.848946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:48.848957 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:48.849008 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:48.850191 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.850714 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.851710 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.851912 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.851969 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:48.854324 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:48.854325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:48.854325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:48.854333 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:48.855611 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:48.856183 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:48.856184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:48.856184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:48.856247 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:48.856639 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:48.856640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:48.856641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:48.856774 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:48.856843 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:48.857199 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 05:51:48.857285 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 05:51:48.857465 32177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:48.847997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.3:47153)
0926 05:51:48.848232 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:48.848285 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:48.848832 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:48.849094 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:48.849099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:48.849850 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:48.850059 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:48.850064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.850809 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.850810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:48.851614 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:48.851616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:48.851989 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:48.851990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:48.852131 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:48.852681 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:48.852690 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:48.854204 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:48.855717 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:48.855726 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:48.855728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:48.856014 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:48.856338 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:48.856339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:48.856339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:48.856517 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:48.856882 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:48.856888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:48.857034 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 05:51:48.857137 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 05:51:48.857283 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
I20260926 05:51:48.858021 32175 heartbeater.cc:347] Connected to a master server at 127.24.255.62:46751
I20260926 05:51:48.858161 32175 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:48.858350 32175 heartbeater.cc:511] Master 127.24.255.62:46751 requested a full tablet report, sending...
I20260926 05:51:48.858824 31716 ts_manager.cc:194] Registered new tserver with Master: e0390f7fabb244bc902af3b9eab49ded (127.24.255.3:46849)
I20260926 05:51:48.859279 31716 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:47153
I20260926 05:51:48.859705 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.255.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.255.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.255.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.61.keytab.
Entry for principal oryx/127.24.255.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.61.keytab.
I20260926 05:51:48.885092 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.61:38879
--webserver_interface=127.24.255.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.61.keytab
--principal=oryx/127.24.255.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:46751,127.24.255.61:38879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:48.970644 32180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:48.970872 32180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:48.970898 32180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:48.972148 32180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:48.972196 32180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:48.972209 32180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:48.972220 32180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:48.972231 32180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:48.973793 32180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:46751,127.24.255.61:38879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.61.keytab
--principal=oryx/127.24.255.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.61:38879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.255.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
W20260926 05:51:48.973910 32180 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.255.62:46751,127.24.255.61:38879'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 05:51:48.974078 32180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:48.974345 32180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:48.979602 32188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:48.979591 32180 server_base.cc:1034] running on GCE node
W20260926 05:51:48.979591 32186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:48.979591 32185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:48.980026 32180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:48.980329 32180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:48.981559 32180 hybrid_clock.cc:648] HybridClock initialized: now 1790401908981536 us; error 39 us; skew 500 ppm
Sep 26 05:51:48 dist-test-slave-7fr8 krb5kdc[31671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:48.984229 32180 init.cc:377] Logged in from keytab as oryx/127.24.255.61@KRBTEST.COM (short username oryx)
I20260926 05:51:48.984751 32180 webserver.cc:460] Webserver started at http://127.24.255.61:44899/ using document root <none> and password file <none>
I20260926 05:51:48.984978 32180 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:48.985026 32180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:48.985157 32180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:48.986279 32180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/data/instance:
uuid: "d20eeed10ddf459cbcffe8ec5d9579f4"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "4b202d6da19688e0dad427bbe9411273"
server_key_iv: "cc4f0149f1b6b1811301d0a43bbb2da2"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.986615 32180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/wal/instance:
uuid: "d20eeed10ddf459cbcffe8ec5d9579f4"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "4b202d6da19688e0dad427bbe9411273"
server_key_iv: "cc4f0149f1b6b1811301d0a43bbb2da2"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.987973 32180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:48.988795 32195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:48.989025 32180 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:48.989114 32180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/wal
uuid: "d20eeed10ddf459cbcffe8ec5d9579f4"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "4b202d6da19688e0dad427bbe9411273"
server_key_iv: "cc4f0149f1b6b1811301d0a43bbb2da2"
server_key_version: "encryptionkey@0"
I20260926 05:51:48.989180 32180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:49.003262 32180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:49.005338 32180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:49.005493 32180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:49.009341 32180 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.61:38879
I20260926 05:51:49.009443 32247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.61:38879 every 8 connection(s)
I20260926 05:51:49.009732 32180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/data/info.pb
I20260926 05:51:49.010298 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 32180
I20260926 05:51:49.010795 32248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:49.013056 32248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.255.61@KRBTEST.COM
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.61@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:49.020980 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.010939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.24.255.61:38879)
0926 05:51:49.011097 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.011104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.011129 (+ 25us) server_negotiation.cc:407] Connection header received
0926 05:51:49.011609 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.011616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.011664 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.011779 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.012519 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.013475 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.014541 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.014974 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.015086 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.017613 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.017619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.017620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.017627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.019193 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.019660 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.019663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.019663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.019747 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.020004 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.020005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.020005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.020315 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.020418 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.020637 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.020742 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260926 05:51:49.021302 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.013781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.24.255.62:46751)
0926 05:51:49.013839 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.013840 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.013953 (+ 113us) server_negotiation.cc:407] Connection header received
0926 05:51:49.014040 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.014041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.014050 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.014118 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.014639 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.015109 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.016024 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.016214 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.016271 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.018398 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.018400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.018400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.018408 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.019829 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.020256 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.020257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.020257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.020389 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.020605 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.020606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.020606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.020804 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.020864 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.021124 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.021218 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:51:49.021406 32252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.013681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:57434)
0926 05:51:49.013840 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.013884 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.013958 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.014144 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.014147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.014473 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.014559 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.014561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.015203 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.015204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.015937 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.015940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.016294 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.016296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.016367 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.016903 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.016914 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.018308 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.019915 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.019918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.019920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.020167 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.020473 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.020473 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.020473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.020538 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.020859 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.020864 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.020910 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.020999 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.021302 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260926 05:51:49.022140 32248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.61@KRBTEST.COM for oryx/127.24.255.61@KRBTEST.COM
I20260926 05:51:49.028421 32252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.022412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38879 (local address 127.0.0.1:58464)
0926 05:51:49.022459 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.022469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.022506 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.022606 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.022606 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.022735 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.022781 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.022783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.023373 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.023374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.023837 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.023838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.024174 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.024175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.024236 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.024725 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.024729 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.025951 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.027497 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.027498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.027498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.027674 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.027849 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.027850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.027850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.027897 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.028121 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.028122 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.028160 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.028218 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.028322 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 05:51:49.028774 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.022478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.24.255.61:38879)
0926 05:51:49.022511 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.022513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.022533 (+ 20us) server_negotiation.cc:407] Connection header received
0926 05:51:49.022549 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.022549 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.022557 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.022598 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.022846 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.023307 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.023925 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.024116 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.024169 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.026036 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.026038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.026038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.026043 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.027368 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.027744 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.027745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.027745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.027792 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.027948 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.027949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.027949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.028063 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.028137 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.028246 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.028421 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":19}
I20260926 05:51:49.030251 32248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: Bootstrap starting.
I20260926 05:51:49.031028 32248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:49.031255 32248 log.cc:824] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:49.031931 32248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: No bootstrap required, opened a new log
I20260926 05:51:49.033286 32248 raft_consensus.cc:348] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } }
I20260926 05:51:49.033416 32248 raft_consensus.cc:374] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:49.033439 32248 raft_consensus.cc:729] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20eeed10ddf459cbcffe8ec5d9579f4, State: Initialized, Role: FOLLOWER
I20260926 05:51:49.033536 32248 consensus_queue.cc:260] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } }
I20260926 05:51:49.033774 32254 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } } }
I20260926 05:51:49.033826 32248 sys_catalog.cc:564] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:49.033850 32254 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:51:49.035465 32265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:51:49.035567 32265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:49.037356 32180 master_runner.cc:428] Detected that this master d20eeed10ddf459cbcffe8ec5d9579f4 is joining an existing cluster
I20260926 05:51:49.037406 32180 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.255.61:38879
I20260926 05:51:49.045146 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.038906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.24.255.61:38879)
0926 05:51:49.038941 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.038943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.039054 (+ 111us) server_negotiation.cc:407] Connection header received
0926 05:51:49.039111 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.039111 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.039120 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.039164 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.039617 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.040342 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.041136 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.041334 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.041388 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.042447 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.042448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.042449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.042456 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.043961 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.044380 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.044382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.044382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.044439 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.044641 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.044641 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.044642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.044769 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.044845 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.044968 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.045042 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":20}
I20260926 05:51:49.045605 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.038858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38879 (local address 127.0.0.1:58476)
0926 05:51:49.038987 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.038996 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.039055 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.039201 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.039202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.039460 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.039529 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.039531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.040425 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.040426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.041047 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.041049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.041396 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.041397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.041465 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.041917 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.041922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.042308 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.044060 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.044061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.044062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.044292 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.044514 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.044515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.044515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.044579 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.044830 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.044831 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.044871 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.045383 (+ 512us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.045500 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260926 05:51:49.045758 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.038690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.24.255.62:46751)
0926 05:51:49.038749 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.038751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.039056 (+ 305us) server_negotiation.cc:407] Connection header received
0926 05:51:49.039167 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.039168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.039179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.039223 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.039643 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.040374 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.041431 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.042060 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.042133 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.043187 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.043188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.043188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.043196 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.044617 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.044956 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.044956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.044957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.045016 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.045209 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.045210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.045210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.045345 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.045406 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.045591 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.045672 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:51:49.046005 32271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.038620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:57448)
0926 05:51:49.038986 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.038997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.039084 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.039241 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.039242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.039506 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.039577 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.039579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.040456 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.040457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.041329 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.041332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.042211 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.042212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.042287 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.042676 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.042681 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.043088 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.044680 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.044680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.044681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.044882 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.045082 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.045083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.045083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.045146 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.045431 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.045431 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.045482 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.045828 (+ 346us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.045935 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":85,"threads_started":1}
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.62@KRBTEST.COM for oryx/127.24.255.61@KRBTEST.COM
I20260926 05:51:49.057119 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.048466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38879 (local address 127.0.0.1:58478)
0926 05:51:49.048619 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.048662 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.048741 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.048970 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.048974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.049336 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.049400 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.049403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.050075 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.050076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.051105 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.051107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.051466 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.051467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.051561 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.052043 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.052048 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.053728 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.055789 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.055800 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.055803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.056051 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.056284 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.056285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.056286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.056358 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.056754 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.056762 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.056821 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.056902 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.057036 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260926 05:51:49.057137 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.048630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.24.255.61:38879)
0926 05:51:49.048683 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.048685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.048715 (+ 30us) server_negotiation.cc:407] Connection header received
0926 05:51:49.048884 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.048885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.048894 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.048948 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.049541 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.049993 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.051200 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.051389 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.051448 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.053833 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.053835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.053835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.053843 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.055700 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.056129 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.056130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.056130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.056195 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.056476 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.056477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.056477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.056620 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.056694 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.056941 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.057041 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:51:49.058075 31716 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.255.61:38879: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4"
member_type: NON_VOTER
last_known_addr {
host: "127.24.255.61"
port: 38879
}
attrs {
promote: true
}
}
dest_uuid: "59ebed1574644d97a63c34f3b3107cef"
cas_config_opid_index: -1
I20260926 05:51:49.058447 31716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true } }
I20260926 05:51:49.059000 32274 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d20eeed10ddf459cbcffe8ec5d9579f4 (127.24.255.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true } } }
I20260926 05:51:49.059330 32222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:49.060192 32274 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.255.61:38879
I20260926 05:51:49.060204 32222 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 FOLLOWER]: Refusing update from remote peer 59ebed1574644d97a63c34f3b3107cef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 05:51:49.060513 32275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true } } }
I20260926 05:51:49.060581 32180 master.cc:537] Master@127.24.255.61:38879 shutting down...
I20260926 05:51:49.060616 32275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:49.060743 32275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 05:51:49.061404 31703 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 05:51:49.062467 31703 consensus_peers.cc:487] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef -> Peer d20eeed10ddf459cbcffe8ec5d9579f4 (127.24.255.61:38879): Couldn't send request to peer d20eeed10ddf459cbcffe8ec5d9579f4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 05:51:49.063191 32180 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:49.063280 32180 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:49.063333 32180 tablet_replica.cc:330] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: stopping tablet replica
I20260926 05:51:49.077035 32180 master.cc:552] Master@127.24.255.61:38879 shutdown complete.
I20260926 05:51:49.077105 32180 master_runner.cc:305] Clearing existing system tablet
I20260926 05:51:49.077231 32180 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:49.077278 32180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:49.078054 32180 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:49.078647 32279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.078823 32180 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:49.078897 32180 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:51:49.078918 32180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/wal
uuid: "d20eeed10ddf459cbcffe8ec5d9579f4"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "4b202d6da19688e0dad427bbe9411273"
server_key_iv: "cc4f0149f1b6b1811301d0a43bbb2da2"
server_key_version: "encryptionkey@0"
I20260926 05:51:49.079686 32180 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:49.081029 32180 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 05:51:49.081090 32180 log.cc:1196] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 05:51:49.081256 32180 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: Deleting consensus metadata
I20260926 05:51:49.081365 32180 master_runner.cc:315] Copying system tablet from 127.24.255.62:46751
I20260926 05:51:49.081875 32180 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: tablet copy: Beginning tablet copy session from remote peer at address 127.24.255.62:46751
I20260926 05:51:49.088848 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.082215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:57460)
0926 05:51:49.082328 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.082338 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.082393 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.082537 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.082538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.082693 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.082754 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.082756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.083730 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.083731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.084332 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.084334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.084858 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.084862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.084924 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.085413 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.085417 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.085752 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.087688 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.087689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.087689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.087869 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.088129 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.088130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.088131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.088196 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.088534 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.088535 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.088582 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.088660 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.088761 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260926 05:51:49.088892 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.082266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.24.255.62:46751)
0926 05:51:49.082317 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.082319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.082385 (+ 66us) server_negotiation.cc:407] Connection header received
0926 05:51:49.082439 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.082439 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.082450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.082533 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.082825 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.083634 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.084419 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.084725 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.084796 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.085844 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.085845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.085846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.085853 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.087574 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.087951 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.087952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.087953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.088038 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.088269 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.088270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.088270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.088428 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.088509 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.088702 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.088796 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:51:49.089534 31736 tablet_copy_service.cc:140] P 59ebed1574644d97a63c34f3b3107cef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d20eeed10ddf459cbcffe8ec5d9579f4 ({username='oryx', principal='oryx/127.24.255.61@KRBTEST.COM'} at 127.0.0.1:57460)
I20260926 05:51:49.089629 31736 tablet_copy_service.cc:161] P 59ebed1574644d97a63c34f3b3107cef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d20eeed10ddf459cbcffe8ec5d9579f4 at {username='oryx', principal='oryx/127.24.255.61@KRBTEST.COM'} at 127.0.0.1:57460: session id = d20eeed10ddf459cbcffe8ec5d9579f4-00000000000000000000000000000000
I20260926 05:51:49.090319 31736 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 05:51:49.090894 32180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 05:51:49.091874 32180 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: tablet copy: Starting download of 0 data blocks...
I20260926 05:51:49.092001 32180 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: tablet copy: Starting download of 1 WAL segments...
I20260926 05:51:49.092936 32180 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 05:51:49.093786 31736 tablet_copy_service.cc:342] P 59ebed1574644d97a63c34f3b3107cef: Request end of tablet copy session d20eeed10ddf459cbcffe8ec5d9579f4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.255.61@KRBTEST.COM'} at 127.0.0.1:57460
I20260926 05:51:49.093845 31736 tablet_copy_service.cc:433] P 59ebed1574644d97a63c34f3b3107cef: ending tablet copy session d20eeed10ddf459cbcffe8ec5d9579f4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d20eeed10ddf459cbcffe8ec5d9579f4
I20260926 05:51:49.096153 32180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:49.097486 32287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:49.097625 32180 server_base.cc:1034] running on GCE node
W20260926 05:51:49.097543 32286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:49.097630 32289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:49.098376 32180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:49.098565 32180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:49.099706 32180 hybrid_clock.cc:648] HybridClock initialized: now 1790401909099696 us; error 34 us; skew 500 ppm
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:49.101193 32180 init.cc:377] Logged in from keytab as oryx/127.24.255.61@KRBTEST.COM (short username oryx)
I20260926 05:51:49.101481 32180 webserver.cc:460] Webserver started at http://127.24.255.61:32799/ using document root <none> and password file <none>
I20260926 05:51:49.101593 32180 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:49.101624 32180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:49.102378 32180 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:49.103067 32296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.103189 32180 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:49.103241 32180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/wal
uuid: "d20eeed10ddf459cbcffe8ec5d9579f4"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "4b202d6da19688e0dad427bbe9411273"
server_key_iv: "cc4f0149f1b6b1811301d0a43bbb2da2"
server_key_version: "encryptionkey@0"
I20260926 05:51:49.103296 32180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:49.125916 32180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:49.127812 32180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:49.130673 32180 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.61:38879
I20260926 05:51:49.130720 32348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.61:38879 every 8 connection(s)
I20260926 05:51:49.131121 32180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-1/data/info.pb
I20260926 05:51:49.131812 32349 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:51:49.132234 32349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: Bootstrap starting.
I20260926 05:51:49.134651 32349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:51:49.134869 32349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: Bootstrap complete.
I20260926 05:51:49.135092 32349 raft_consensus.cc:348] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true } }
I20260926 05:51:49.135188 32349 raft_consensus.cc:729] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d20eeed10ddf459cbcffe8ec5d9579f4, State: Initialized, Role: LEARNER
I20260926 05:51:49.135216 32349 consensus_queue.cc:260] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true } }
I20260926 05:51:49.135418 32349 sys_catalog.cc:564] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:49.135406 32352 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true } } }
I20260926 05:51:49.135535 32352 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: This master's current role is: LEARNER
I20260926 05:51:49.136821 32363 catalog_manager.cc:1259] Loaded cluster ID: 76801b8588544dd9add52d55f500263a
I20260926 05:51:49.136874 32363 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: loading cluster ID for follower catalog manager: success
I20260926 05:51:49.137889 32363 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: acquiring CA information for follower catalog manager: success
I20260926 05:51:49.138219 32363 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:51:49.181221 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.173365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.24.255.61:38879)
0926 05:51:49.173513 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.173515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.173523 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:49.173546 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.173547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.173556 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.173619 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.174214 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.175209 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.175935 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.176134 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.176189 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.177526 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.177527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.177528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.177534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.179716 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.180113 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.180114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.180115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.180178 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.180334 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.180335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.180335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.180469 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.180573 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.181003 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.181107 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for oryx/127.24.255.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.255.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.255.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.60.keytab.
Entry for principal oryx/127.24.255.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.60.keytab.
I20260926 05:51:49.209491 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.60:35029
--webserver_interface=127.24.255.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.60.keytab
--principal=oryx/127.24.255.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:46751,127.24.255.61:38879,127.24.255.60:35029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:49.298465 32367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:49.298704 32367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:49.298730 32367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:49.301753 32367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:49.301837 32367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:49.301853 32367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:49.301867 32367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:49.301877 32367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:49.304731 32367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:46751,127.24.255.61:38879,127.24.255.60:35029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.60.keytab
--principal=oryx/127.24.255.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.60:35029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.255.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:49.305044 32367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:49.305434 32367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:49.311337 32375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:49.311338 32372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:49.311750 32373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:49.312368 32367 server_base.cc:1034] running on GCE node
I20260926 05:51:49.312711 32367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:49.313059 32367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:49.314258 32367 hybrid_clock.cc:648] HybridClock initialized: now 1790401909314241 us; error 34 us; skew 500 ppm
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:49.317196 32367 init.cc:377] Logged in from keytab as oryx/127.24.255.60@KRBTEST.COM (short username oryx)
I20260926 05:51:49.318421 32367 webserver.cc:460] Webserver started at http://127.24.255.60:44703/ using document root <none> and password file <none>
I20260926 05:51:49.318709 32367 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:49.318770 32367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:49.318871 32367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:49.319970 32367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/data/instance:
uuid: "ebe661be221845cd986e688e7b4f6041"
format_stamp: "Formatted at 2026-09-26 05:51:49 on dist-test-slave-7fr8"
server_key: "9ea277b120e1c72c7618fe33f8c9a5fa"
server_key_iv: "094a2b94d46afe31fc80dbec947cd302"
server_key_version: "encryptionkey@0"
I20260926 05:51:49.320288 32367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/wal/instance:
uuid: "ebe661be221845cd986e688e7b4f6041"
format_stamp: "Formatted at 2026-09-26 05:51:49 on dist-test-slave-7fr8"
server_key: "9ea277b120e1c72c7618fe33f8c9a5fa"
server_key_iv: "094a2b94d46afe31fc80dbec947cd302"
server_key_version: "encryptionkey@0"
I20260926 05:51:49.321851 32367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:49.323235 32382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.323518 32367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:49.323607 32367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/wal
uuid: "ebe661be221845cd986e688e7b4f6041"
format_stamp: "Formatted at 2026-09-26 05:51:49 on dist-test-slave-7fr8"
server_key: "9ea277b120e1c72c7618fe33f8c9a5fa"
server_key_iv: "094a2b94d46afe31fc80dbec947cd302"
server_key_version: "encryptionkey@0"
I20260926 05:51:49.323673 32367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:49.334918 32367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:49.337772 32367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:49.337903 32367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:49.341929 32367 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.60:35029
I20260926 05:51:49.341962 32434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.60:35029 every 8 connection(s)
I20260926 05:51:49.342376 32367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/data/info.pb
I20260926 05:51:49.343415 32435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:49.345240 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 32367
I20260926 05:51:49.345453 32435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.60@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.255.60@KRBTEST.COM
I20260926 05:51:49.355612 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.346270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57476 (local address 127.24.255.62:46751)
0926 05:51:49.346342 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.346344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.346467 (+ 123us) server_negotiation.cc:407] Connection header received
0926 05:51:49.347466 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.347467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.347479 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.347531 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.348008 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.348499 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.349506 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.349703 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.349761 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.352240 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.352242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.352242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.352252 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.353890 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.354570 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.354571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.354572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.354635 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.354931 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.354932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.354933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.355059 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.355113 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.355431 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.355514 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 05:51:49.355885 32437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.346027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.24.255.60:35029)
0926 05:51:49.346230 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.346240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.346272 (+ 32us) server_negotiation.cc:407] Connection header received
0926 05:51:49.346720 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.346727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.346749 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.346849 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.347505 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.348446 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.349407 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.349728 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.349871 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.352654 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.352663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.352663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.352670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.354602 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.354966 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.354969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.354969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.355030 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.355214 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.355215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.355215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.355363 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.355500 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.355601 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.355683 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":119,"threads_started":1}
I20260926 05:51:49.356117 32438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.346145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:57476)
0926 05:51:49.346348 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.346389 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.346749 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.347563 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.347566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.347819 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.347913 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.347917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.348594 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.348595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.349402 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.349405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.349812 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.349813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.349874 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.350431 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.350442 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.352119 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.354016 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.354022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.354024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.354458 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.354729 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.354731 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.354731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.354848 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.355119 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.355125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.355292 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.355376 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.355539 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260926 05:51:49.356647 32435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.60@KRBTEST.COM for oryx/127.24.255.61@KRBTEST.COM
I20260926 05:51:49.365127 32438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.357024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38879 (local address 127.0.0.1:58510)
0926 05:51:49.357115 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.357129 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.357176 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.357789 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.357791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.357990 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.358072 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.358075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.359250 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.359252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.359888 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.359891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.360269 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.360269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.360348 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.360987 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.360992 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.362311 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.364094 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.364095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.364096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.364267 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.364454 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.364454 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.364455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.364534 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.364791 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.364792 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.364840 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.364900 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.365024 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 05:51:49.365126 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.357264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.24.255.61:38879)
0926 05:51:49.357442 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.357445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.357464 (+ 19us) server_negotiation.cc:407] Connection header received
0926 05:51:49.357575 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.357577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.357595 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.357706 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.358338 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.358958 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.359985 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.360195 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.360276 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.362415 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.362418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.362418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.362426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.363945 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.364339 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.364340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.364340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.364394 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.364605 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.364606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.364606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.364730 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.364787 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.364933 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.365022 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260926 05:51:49.365695 32435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.60@KRBTEST.COM for oryx/127.24.255.60@KRBTEST.COM
I20260926 05:51:49.372465 32438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.365971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35029 (local address 127.0.0.1:55346)
0926 05:51:49.366003 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.366020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.366055 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.366227 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.366227 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.366340 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.366379 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.366380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.367160 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.367161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.367670 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.367671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.368154 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.368155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.368198 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.368657 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.368661 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.369935 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.371380 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.371381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.371381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.371558 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.371790 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.371791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.371792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.371847 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.372139 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.372141 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.372184 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.372249 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.372346 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":2}
I20260926 05:51:49.372476 32437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.366061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.24.255.60:35029)
0926 05:51:49.366128 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.366130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.366138 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:49.366160 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.366161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.366171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.366225 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.366491 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.367045 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.367843 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.368062 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.368128 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.370043 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.370045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.370045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.370052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.371287 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.371663 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.371664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.371665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.371721 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.371930 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.371931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.371931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.372047 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.372110 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.372265 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.372402 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":46}
I20260926 05:51:49.374068 32435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: Bootstrap starting.
I20260926 05:51:49.374933 32435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:49.375207 32435 log.cc:824] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:49.375958 32435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: No bootstrap required, opened a new log
I20260926 05:51:49.377389 32435 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } }
I20260926 05:51:49.377552 32435 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:49.377588 32435 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe661be221845cd986e688e7b4f6041, State: Initialized, Role: FOLLOWER
I20260926 05:51:49.377702 32435 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } }
I20260926 05:51:49.377970 32440 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } } }
I20260926 05:51:49.378034 32435 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:49.378057 32440 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:51:49.379792 32451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:51:49.379850 32451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:49.382432 32367 master_runner.cc:428] Detected that this master ebe661be221845cd986e688e7b4f6041 is joining an existing cluster
I20260926 05:51:49.382500 32367 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.255.60:35029
I20260926 05:51:49.391337 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.383914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38879 (local address 127.0.0.1:58512)
0926 05:51:49.384221 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.384229 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.384295 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.384443 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.384443 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.384701 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.384796 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.384799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.385761 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.385762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.386642 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.386644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.387166 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.387167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.387232 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.387683 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.387689 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.388183 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.390021 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.390022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.390023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.390251 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.390508 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.390509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.390509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.390575 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.391009 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.391011 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.391060 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.391124 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.391241 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":75,"threads_started":1}
I20260926 05:51:49.391359 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.383768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.24.255.61:38879)
0926 05:51:49.383811 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.383813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.384273 (+ 460us) server_negotiation.cc:407] Connection header received
0926 05:51:49.384349 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.384350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.384361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.384422 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.384927 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.385675 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.386731 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.387035 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.387074 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.388300 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.388302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.388303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.388311 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.389898 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.390337 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.390338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.390338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.390408 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.390750 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.390751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.390752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.390904 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.390982 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.391165 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.391259 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:51:49.407269 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 31778
I20260926 05:51:49.412412 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.383724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.24.255.62:46751)
0926 05:51:49.383806 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.383808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.383984 (+ 176us) server_negotiation.cc:407] Connection header received
0926 05:51:49.384043 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.384044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.384057 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.384113 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.384815 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.385597 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.386759 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.387070 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.387113 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.408292 (+ 21179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.408294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.408294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.408304 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.411146 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.411631 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.411632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.411633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.411696 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.411867 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.411868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.411868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.412005 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.412045 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.412229 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.412315 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 05:51:49.412446 32437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.383766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.24.255.60:35029)
0926 05:51:49.383834 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.383836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.384094 (+ 258us) server_negotiation.cc:407] Connection header received
0926 05:51:49.384170 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.384171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.384179 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.384222 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.385199 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.385912 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.386847 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.387097 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.387130 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.408122 (+ 20992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.408124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.408125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.408132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.411012 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.411621 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.411622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.411622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.411677 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.411839 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.411840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.411855 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.412012 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.412062 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.412274 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.412359 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 05:51:49.412744 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.383635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:57480)
0926 05:51:49.383899 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.383909 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.383976 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.384150 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.384151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.384649 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.384717 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.384719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.385698 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.385698 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.386657 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.386659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.387190 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.387191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.387255 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.407590 (+ 20335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.407595 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.408156 (+ 561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.411246 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.411247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.411248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.411543 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.411758 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.411758 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.411759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.411810 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.412086 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.412086 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.412131 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.412521 (+ 390us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.412632 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":66,"threads_started":1}
I20260926 05:51:49.412860 32457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.383636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35029 (local address 127.0.0.1:55350)
0926 05:51:49.384037 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.384046 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.384102 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.384257 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.384257 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.384535 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.385126 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.385128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.385981 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.385981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.386781 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.386783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.387203 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.387203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.387255 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.406800 (+ 19545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.406808 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.407443 (+ 635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.411120 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.411121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.411121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.411528 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.411737 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.411738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.411738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.411785 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.412112 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.412113 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.412155 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.412659 (+ 504us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.412760 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":15,"thread_start_us":93,"threads_started":1}
I20260926 05:51:49.414412 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:39449
--local_ip_for_outbound_sockets=127.24.255.1
--tserver_master_addrs=127.24.255.62:46751
--webserver_port=43339
--webserver_interface=127.24.255.1
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab
--principal=oryx/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.255.62:46751,127.24.255.61:38879
--tserver_master_addrs=127.24.255.62:46751,127.24.255.61:38879,127.24.255.60:35029 with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401908, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.62@KRBTEST.COM for oryx/127.24.255.60@KRBTEST.COM
I20260926 05:51:49.423707 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.415488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35029 (local address 127.0.0.1:55358)
0926 05:51:49.415541 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.415551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.415589 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.415756 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.415758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.415934 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.415985 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.415987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.416836 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.416838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.417711 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.417715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.418229 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.418230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.418297 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.418872 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.418878 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.420449 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.422335 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.422337 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.422338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.422713 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.422958 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.422958 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.422959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.423026 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.423345 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.423346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.423398 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.423463 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.423601 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:51:49.424015 32437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.415541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55358 (local address 127.24.255.60:35029)
0926 05:51:49.415574 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.415576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.415584 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:49.415647 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.415648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.415658 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.415702 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.416063 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.416742 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.417800 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.418055 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.418175 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.420580 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.420582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.420582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.420590 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.422226 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.422820 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.422821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.422821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.422888 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.423092 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.423093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.423093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.423257 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.423319 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.423850 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.423931 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:51:49.424523 31716 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.255.60:35029: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ebe661be221845cd986e688e7b4f6041"
member_type: NON_VOTER
last_known_addr {
host: "127.24.255.60"
port: 35029
}
attrs {
promote: true
}
}
dest_uuid: "59ebed1574644d97a63c34f3b3107cef"
cas_config_opid_index: 5
I20260926 05:51:49.424798 31716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } }
I20260926 05:51:49.425175 32274 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ebe661be221845cd986e688e7b4f6041 (127.24.255.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } } }
I20260926 05:51:49.425684 32409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:49.426236 32274 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.255.60:35029
I20260926 05:51:49.426316 32274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } } }
I20260926 05:51:49.426383 32274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:49.426879 32367 master.cc:537] Master@127.24.255.60:35029 shutting down...
I20260926 05:51:49.427012 32409 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 1 FOLLOWER]: Refusing update from remote peer 59ebed1574644d97a63c34f3b3107cef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 05:51:49.427371 32275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 05:51:49.428401 31700 consensus_peers.cc:487] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef -> Peer ebe661be221845cd986e688e7b4f6041 (127.24.255.60:35029): Couldn't send request to peer ebe661be221845cd986e688e7b4f6041. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 05:51:49.430132 32367 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:49.430241 32367 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:49.430284 32367 tablet_replica.cc:330] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: stopping tablet replica
I20260926 05:51:49.434666 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.425352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.24.255.61:38879)
0926 05:51:49.428063 (+ 2711us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.428066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.428545 (+ 479us) server_negotiation.cc:407] Connection header received
0926 05:51:49.428659 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.428661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.428686 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.428730 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:49.429048 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.429850 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.433123 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.433775 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.433829 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.433922 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.434521 (+ 599us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.434578 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2660}
I20260926 05:51:49.434969 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.425251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38879 (local address 127.0.0.1:58518)
0926 05:51:49.428489 (+ 3238us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.428501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.428576 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.428908 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.428909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.428911 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:49.428963 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.428965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.430001 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.430002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.432922 (+ 2920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.432930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.434203 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.434204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.434293 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.434338 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.434838 (+ 500us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.434896 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3219}
I20260926 05:51:49.435261 32323 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 LEARNER]: Refusing update from remote peer 59ebed1574644d97a63c34f3b3107cef: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 05:51:49.435475 32275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 05:51:49.435789 32367 master.cc:552] Master@127.24.255.60:35029 shutdown complete.
I20260926 05:51:49.435895 32367 master_runner.cc:305] Clearing existing system tablet
I20260926 05:51:49.436108 32367 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:49.436290 32367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:49.436239 32323 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ebe661be221845cd986e688e7b4f6041 (127.24.255.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } } }
I20260926 05:51:49.437446 32367 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:51:49.437443 32352 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59ebed1574644d97a63c34f3b3107cef. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } } }
I20260926 05:51:49.437537 32352 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: This master's current role is: LEARNER
I20260926 05:51:49.437566 32352 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: NON_VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: true } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } } }
I20260926 05:51:49.437602 32352 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: This master's current role is: LEARNER
I20260926 05:51:49.438170 32464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.438321 32367 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:49.438373 32367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:51:49.438395 32367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/wal
uuid: "ebe661be221845cd986e688e7b4f6041"
format_stamp: "Formatted at 2026-09-26 05:51:49 on dist-test-slave-7fr8"
server_key: "9ea277b120e1c72c7618fe33f8c9a5fa"
server_key_iv: "094a2b94d46afe31fc80dbec947cd302"
server_key_version: "encryptionkey@0"
I20260926 05:51:49.439263 32367 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:49.440972 32367 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 05:51:49.441069 32367 log.cc:1196] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 05:51:49.441293 32367 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: Deleting consensus metadata
I20260926 05:51:49.441433 32367 master_runner.cc:315] Copying system tablet from 127.24.255.62:46751
I20260926 05:51:49.442163 32367 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: tablet copy: Beginning tablet copy session from remote peer at address 127.24.255.62:46751
I20260926 05:51:49.446691 32275 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef: attempting to promote NON_VOTER d20eeed10ddf459cbcffe8ec5d9579f4 to VOTER
I20260926 05:51:49.446959 32275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } }
I20260926 05:51:49.447945 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.447670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35029 (local address 127.0.0.1:55364)
0926 05:51:49.447842 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.447904 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260926 05:51:49.448079 32323 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 LEARNER]: Refusing update from remote peer 59ebed1574644d97a63c34f3b3107cef: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 05:51:49.448297 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.448217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35029 (local address 127.0.0.1:55368)
0926 05:51:49.448251 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.448273 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 05:51:49.448369 31700 consensus_peers.cc:487] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef -> Peer ebe661be221845cd986e688e7b4f6041 (127.24.255.60:35029): Couldn't send request to peer ebe661be221845cd986e688e7b4f6041. Status: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 05:51:49.448452 32274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 05:51:49.449177 32274 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d20eeed10ddf459cbcffe8ec5d9579f4 (127.24.255.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } } }
I20260926 05:51:49.450187 32274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } } }
I20260926 05:51:49.450264 32274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:49.450314 32274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } } }
I20260926 05:51:49.450351 32274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:49.450865 32323 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d20eeed10ddf459cbcffe8ec5d9579f4 (127.24.255.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } } }
I20260926 05:51:49.450989 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.442593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.24.255.62:46751)
0926 05:51:49.442645 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.442648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.442717 (+ 69us) server_negotiation.cc:407] Connection header received
0926 05:51:49.442818 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.442819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.442830 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.442917 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.443351 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.444143 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.445212 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.445511 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.445584 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.446835 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.446837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.446838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.446848 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.449283 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.449747 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.449748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.449749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.449833 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.450057 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.450058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.450058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.450247 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.450556 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.450813 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.450904 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:51:49.451308 32470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.442520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:57488)
0926 05:51:49.442657 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.442668 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.442737 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.442946 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.442948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.443171 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.443251 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.443255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.444247 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.444248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.445105 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.445108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.445598 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.445600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.445673 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.446250 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.446256 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.446690 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.449405 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.449406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.449407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.449653 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.449913 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.449914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.449914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.449979 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.450627 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.450628 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.450691 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.451087 (+ 396us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.451212 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260926 05:51:49.451520 31736 tablet_copy_service.cc:140] P 59ebed1574644d97a63c34f3b3107cef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ebe661be221845cd986e688e7b4f6041 ({username='oryx', principal='oryx/127.24.255.60@KRBTEST.COM'} at 127.0.0.1:57488)
I20260926 05:51:49.451565 31736 tablet_copy_service.cc:161] P 59ebed1574644d97a63c34f3b3107cef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ebe661be221845cd986e688e7b4f6041 at {username='oryx', principal='oryx/127.24.255.60@KRBTEST.COM'} at 127.0.0.1:57488: session id = ebe661be221845cd986e688e7b4f6041-00000000000000000000000000000000
I20260926 05:51:49.451843 32463 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } } }
I20260926 05:51:49.451944 32463 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:49.452153 31736 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 05:51:49.452625 32367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 05:51:49.454387 32367 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: tablet copy: Starting download of 0 data blocks...
I20260926 05:51:49.454541 32367 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: tablet copy: Starting download of 1 WAL segments...
I20260926 05:51:49.455325 32367 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 05:51:49.456324 31736 tablet_copy_service.cc:342] P 59ebed1574644d97a63c34f3b3107cef: Request end of tablet copy session ebe661be221845cd986e688e7b4f6041-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.255.60@KRBTEST.COM'} at 127.0.0.1:57488
I20260926 05:51:49.456398 31736 tablet_copy_service.cc:433] P 59ebed1574644d97a63c34f3b3107cef: ending tablet copy session ebe661be221845cd986e688e7b4f6041-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ebe661be221845cd986e688e7b4f6041
I20260926 05:51:49.459025 32367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:49.460546 32472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:49.463137 32473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:49.465029 32367 server_base.cc:1034] running on GCE node
W20260926 05:51:49.470287 32475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:49.470515 32367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:49.470698 32367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:49.471868 32367 hybrid_clock.cc:648] HybridClock initialized: now 1790401909471859 us; error 29 us; skew 500 ppm
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:49.473453 32367 init.cc:377] Logged in from keytab as oryx/127.24.255.60@KRBTEST.COM (short username oryx)
I20260926 05:51:49.473728 32367 webserver.cc:460] Webserver started at http://127.24.255.60:41833/ using document root <none> and password file <none>
I20260926 05:51:49.473842 32367 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:49.473875 32367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:49.474658 32367 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:51:49.475274 32482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.475428 32367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:51:49.475479 32367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/wal
uuid: "ebe661be221845cd986e688e7b4f6041"
format_stamp: "Formatted at 2026-09-26 05:51:49 on dist-test-slave-7fr8"
server_key: "9ea277b120e1c72c7618fe33f8c9a5fa"
server_key_iv: "094a2b94d46afe31fc80dbec947cd302"
server_key_version: "encryptionkey@0"
I20260926 05:51:49.475540 32367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:49.500651 32367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:49.503079 32367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:49.506963 32367 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.60:35029
I20260926 05:51:49.507395 32367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/master-2/data/info.pb
I20260926 05:51:49.508471 32535 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:51:49.508996 32535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: Bootstrap starting.
I20260926 05:51:49.510767 32534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.60:35029 every 8 connection(s)
I20260926 05:51:49.512665 32535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:51:49.512885 32535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: Bootstrap complete.
I20260926 05:51:49.513098 32535 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } }
I20260926 05:51:49.513197 32535 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ebe661be221845cd986e688e7b4f6041, State: Initialized, Role: LEARNER
I20260926 05:51:49.513233 32535 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } }
I20260926 05:51:49.513432 32535 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:49.513455 32538 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } } }
I20260926 05:51:49.513684 32538 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: This master's current role is: LEARNER
I20260926 05:51:49.515336 32549 catalog_manager.cc:1259] Loaded cluster ID: 76801b8588544dd9add52d55f500263a
I20260926 05:51:49.515436 32549 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: loading cluster ID for follower catalog manager: success
I20260926 05:51:49.516865 32549 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: acquiring CA information for follower catalog manager: success
I20260926 05:51:49.517194 32549 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 05:51:49.529666 32460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:49.530491 32460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:49.530535 32460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:49.532243 32460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:49.532474 32460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:49.532550 32460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:49.535861 32460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab
--principal=oryx/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:39449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=43339
--tserver_master_addrs=127.24.255.62:46751,127.24.255.61:38879,127.24.255.60:35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:49.536201 32460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:49.536468 32460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:49.543107 32555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:49.543107 32557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:49.543108 32554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:49.543397 32460 server_base.cc:1034] running on GCE node
I20260926 05:51:49.543612 32460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:49.543864 32460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:49.545046 32460 hybrid_clock.cc:648] HybridClock initialized: now 1790401909545026 us; error 41 us; skew 500 ppm
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:49.547760 32460 init.cc:377] Logged in from keytab as oryx/127.24.255.1@KRBTEST.COM (short username oryx)
I20260926 05:51:49.548573 32460 webserver.cc:460] Webserver started at http://127.24.255.1:43339/ using document root <none> and password file <none>
I20260926 05:51:49.548979 32460 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:49.549067 32460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:49.550841 32460 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:51:49.551883 32564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.552377 32460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:49.552503 32460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "e1c39876dc73428d87aadbba9ed9de58"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "04899793c074fee7618634aca44eec04"
server_key_iv: "6a77f9ea74025eb9ae88398261972d94"
server_key_version: "encryptionkey@0"
I20260926 05:51:49.552944 32460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:49.579615 32460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:49.581832 32460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:49.581955 32460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:49.582222 32460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:49.582576 32460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:49.582614 32460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.582644 32460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:49.582661 32460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.589278 32460 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:39449
I20260926 05:51:49.589387 32677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:39449 every 8 connection(s)
I20260926 05:51:49.589720 32460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:51:49.593226 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 32460
I20260926 05:51:49.593425 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 31912
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.61@KRBTEST.COM
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.60@KRBTEST.COM
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:49.601270 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:37079
--local_ip_for_outbound_sockets=127.24.255.2
--tserver_master_addrs=127.24.255.62:46751
--webserver_port=41765
--webserver_interface=127.24.255.2
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab
--principal=oryx/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.255.62:46751,127.24.255.61:38879
--tserver_master_addrs=127.24.255.62:46751,127.24.255.61:38879,127.24.255.60:35029 with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:51:49.602737 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.590645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:51727 (local address 127.24.255.61:38879)
0926 05:51:49.590743 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.590745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.590964 (+ 219us) server_negotiation.cc:407] Connection header received
0926 05:51:49.591546 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.591547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.591558 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.591606 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.592436 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.592912 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.594481 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.594768 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.594808 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.598745 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.598747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.598747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.598757 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.600350 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.601273 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.601274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.601274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.601340 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.601884 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.601885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.601886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.602060 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.602130 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.602532 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.602637 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 05:51:49.602756 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.590559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:53745 (local address 127.24.255.60:35029)
0926 05:51:49.590674 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.590677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.590942 (+ 265us) server_negotiation.cc:407] Connection header received
0926 05:51:49.591484 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.591485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.591499 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.591565 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.592389 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.593394 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.594682 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.594960 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.594999 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.599798 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.599801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.599801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.599810 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.601388 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.601738 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.601739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.601739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.601795 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.601960 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.601960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.601960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.602084 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.602146 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.602532 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.602635 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
I20260926 05:51:49.603168 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.590583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:39581 (local address 127.24.255.62:46751)
0926 05:51:49.590693 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.590708 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.590984 (+ 276us) server_negotiation.cc:407] Connection header received
0926 05:51:49.591565 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.591566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.591581 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.591611 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.592403 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.593206 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.594481 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.594768 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.594807 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.600480 (+ 5673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.600483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.600483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.600493 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.601982 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.602347 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.602348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.602348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.602419 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.602612 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.602613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.602613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.602740 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.602789 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.602996 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.603080 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 05:51:49.603485 32685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.590512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35029 (local address 127.24.255.1:53745)
0926 05:51:49.590846 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.590857 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.591376 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.591698 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.591703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.592038 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.592283 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.592287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.593482 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.593483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.594605 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.594606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.595075 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.595076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.595145 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.595837 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.595849 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.599678 (+ 3829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.601478 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.601479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.601479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.601662 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.601858 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.601859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.601859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.601908 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.602188 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.602189 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.602375 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.603023 (+ 648us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.603232 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":580,"thread_start_us":55,"threads_started":1}
I20260926 05:51:49.603559 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.590510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.1:39581)
0926 05:51:49.590803 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.590849 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.591471 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.592069 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.592069 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.592201 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.592284 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.592290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.593378 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.593379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.594379 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.594381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.594921 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.594922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.595081 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.595958 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.595962 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.599676 (+ 3714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.602055 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.602056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.602056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.602268 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.602490 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.602491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.602492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.602545 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.602822 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.602822 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.602864 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.603300 (+ 436us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.603421 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":76,"threads_started":1}
I20260926 05:51:49.604090 32678 heartbeater.cc:347] Connected to a master server at 127.24.255.62:46751
I20260926 05:51:49.604197 32678 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:49.604215 32679 heartbeater.cc:347] Connected to a master server at 127.24.255.60:35029
I20260926 05:51:49.604259 32679 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:49.604409 32678 heartbeater.cc:511] Master 127.24.255.62:46751 requested a full tablet report, sending...
I20260926 05:51:49.604418 32679 heartbeater.cc:511] Master 127.24.255.60:35029 requested a full tablet report, sending...
I20260926 05:51:49.604420 32683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.590549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38879 (local address 127.24.255.1:51727)
0926 05:51:49.590805 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.590849 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.591423 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.591717 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.591718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.592068 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.592284 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.592290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.593025 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.593027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.594377 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.594380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.594907 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.594908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.595081 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.595855 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.595860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.598638 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.600431 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.600441 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.600444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.601181 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.601651 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.601652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.601652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.601803 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.602174 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.602181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.602374 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.603477 (+ 1103us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.604205 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":4,"thread_start_us":91,"threads_started":1}
I20260926 05:51:49.604813 31716 ts_manager.cc:194] Re-registered known tserver with Master: e1c39876dc73428d87aadbba9ed9de58 (127.24.255.1:39449)
I20260926 05:51:49.604916 32499 ts_manager.cc:194] Registered new tserver with Master: e1c39876dc73428d87aadbba9ed9de58 (127.24.255.1:39449)
I20260926 05:51:49.605070 32680 heartbeater.cc:347] Connected to a master server at 127.24.255.61:38879
I20260926 05:51:49.605123 32680 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:49.605185 32680 heartbeater.cc:511] Master 127.24.255.61:38879 requested a full tablet report, sending...
I20260926 05:51:49.605279 31716 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:39581
I20260926 05:51:49.605542 32313 ts_manager.cc:194] Registered new tserver with Master: e1c39876dc73428d87aadbba9ed9de58 (127.24.255.1:39449)
W20260926 05:51:49.689416 32686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:49.689677 32686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:49.689705 32686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:49.691226 32686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:49.691296 32686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:49.691326 32686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:49.693019 32686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab
--principal=oryx/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:37079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=41765
--tserver_master_addrs=127.24.255.62:46751,127.24.255.61:38879,127.24.255.60:35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:49.693256 32686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:49.693465 32686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:49.699837 32694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:49.699862 32692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:49.699837 32691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:49.699873 32686 server_base.cc:1034] running on GCE node
I20260926 05:51:49.700366 32686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:49.700645 32686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:49.701856 32686 hybrid_clock.cc:648] HybridClock initialized: now 1790401909701843 us; error 40 us; skew 500 ppm
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:49.705014 32686 init.cc:377] Logged in from keytab as oryx/127.24.255.2@KRBTEST.COM (short username oryx)
I20260926 05:51:49.705757 32686 webserver.cc:460] Webserver started at http://127.24.255.2:41765/ using document root <none> and password file <none>
I20260926 05:51:49.706063 32686 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:49.706184 32686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:49.707695 32686 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:49.708323 32701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.708585 32686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:49.708663 32686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "0513b0a4ecb94eb1935b1eda3d179c35"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "1d534e9639b2ebafd99524170b43ffca"
server_key_iv: "f2183880d738b989d81614e22bdec3cd"
server_key_version: "encryptionkey@0"
I20260926 05:51:49.709040 32686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:49.724697 32686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:49.726951 32686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:49.727077 32686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:49.727308 32686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:49.727638 32686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:49.727674 32686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.727705 32686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:49.727730 32686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.733407 32686 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:37079
I20260926 05:51:49.733471 346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:37079 every 8 connection(s)
I20260926 05:51:49.733809 32686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:51:49.738752 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 32686
I20260926 05:51:49.738920 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 32046
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.60@KRBTEST.COM
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.61@KRBTEST.COM
I20260926 05:51:49.745302 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:46849
--local_ip_for_outbound_sockets=127.24.255.3
--tserver_master_addrs=127.24.255.62:46751
--webserver_port=39813
--webserver_interface=127.24.255.3
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab
--principal=oryx/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.255.62:46751,127.24.255.61:38879
--tserver_master_addrs=127.24.255.62:46751,127.24.255.61:38879,127.24.255.60:35029 with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:51:49.746419 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.734821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:47481 (local address 127.24.255.60:35029)
0926 05:51:49.734865 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.734867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.734946 (+ 79us) server_negotiation.cc:407] Connection header received
0926 05:51:49.736180 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.736182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.736197 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.736224 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.737785 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.738372 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.740596 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.740852 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.740915 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.743160 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.743161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.743162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.743170 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.744748 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.745304 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.745305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.745305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.745366 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.745655 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.745655 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.745656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.745788 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.745828 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.746232 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.746328 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:51:49.746654 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.734635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:46071 (local address 127.24.255.61:38879)
0926 05:51:49.734736 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.734738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.734933 (+ 195us) server_negotiation.cc:407] Connection header received
0926 05:51:49.736086 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.736087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.736100 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.736157 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.738190 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.738822 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.740375 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.740573 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.740633 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.743386 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.743388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.743388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.743397 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.744726 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.745305 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.745306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.745306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.745367 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.745655 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.745656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.745656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.745791 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.745830 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.746228 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.746313 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 05:51:49.746989 351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.734531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35029 (local address 127.24.255.2:47481)
0926 05:51:49.734832 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.734841 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.736059 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.736332 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.736335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.736798 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.737054 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.737060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.738477 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.738479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.740487 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.740491 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.740932 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.740933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.741082 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.741499 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.741504 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.743040 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.744838 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.744839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.744840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.745215 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.745429 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.745430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.745430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.745585 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.745875 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.745881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.746053 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.746523 (+ 470us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.746748 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":60,"thread_start_us":93,"threads_started":1}
I20260926 05:51:49.747431 348 heartbeater.cc:347] Connected to a master server at 127.24.255.60:35029
I20260926 05:51:49.747538 348 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:49.747736 348 heartbeater.cc:511] Master 127.24.255.60:35029 requested a full tablet report, sending...
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:49.748200 32499 ts_manager.cc:194] Registered new tserver with Master: 0513b0a4ecb94eb1935b1eda3d179c35 (127.24.255.2:37079)
I20260926 05:51:49.748306 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.734531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38879 (local address 127.24.255.2:46071)
0926 05:51:49.734796 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.734839 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.735978 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.736328 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.736333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.736773 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.737054 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.737060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.739613 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.739614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.740282 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.740284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.740653 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.740653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.740796 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.741425 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.741436 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.743289 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.744815 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.744826 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.744829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.745214 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.745442 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.745442 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.745443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.745585 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.745883 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.745884 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.746053 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.746773 (+ 720us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.748206 (+ 1433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":86,"threads_started":1}
I20260926 05:51:49.749152 349 heartbeater.cc:347] Connected to a master server at 127.24.255.61:38879
I20260926 05:51:49.749209 349 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:49.749284 349 heartbeater.cc:511] Master 127.24.255.61:38879 requested a full tablet report, sending...
I20260926 05:51:49.749593 32313 ts_manager.cc:194] Registered new tserver with Master: 0513b0a4ecb94eb1935b1eda3d179c35 (127.24.255.2:37079)
I20260926 05:51:49.750950 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.734692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:37997 (local address 127.24.255.62:46751)
0926 05:51:49.734770 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.734772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.735053 (+ 281us) server_negotiation.cc:407] Connection header received
0926 05:51:49.736168 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.736170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.736182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.736231 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.737180 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.737689 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.745741 (+ 8052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.746051 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.746122 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.748218 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.748220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.748220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.748229 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.749855 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.750206 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.750207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.750207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.750268 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.750455 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.750456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.750456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.750576 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.750633 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.750786 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.750858 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 05:51:49.750965 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.734530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.2:37997)
0926 05:51:49.734988 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.734996 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.736098 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.736330 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.736335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.736825 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.737054 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.737060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.737876 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.737877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.745640 (+ 7763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.745642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.746203 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.746204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.746271 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.746648 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.746652 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.748118 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.749933 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.749934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.749935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.750137 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.750330 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.750331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.750331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.750389 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.750637 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.750638 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.750682 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.750738 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.750852 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":111,"thread_start_us":73,"threads_started":1}
I20260926 05:51:49.751302 347 heartbeater.cc:347] Connected to a master server at 127.24.255.62:46751
I20260926 05:51:49.751350 347 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:49.751406 347 heartbeater.cc:511] Master 127.24.255.62:46751 requested a full tablet report, sending...
I20260926 05:51:49.751665 31716 ts_manager.cc:194] Re-registered known tserver with Master: 0513b0a4ecb94eb1935b1eda3d179c35 (127.24.255.2:37079)
I20260926 05:51:49.752039 31716 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:37997
W20260926 05:51:49.830828 354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:49.831054 354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:49.831089 354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:49.832330 354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:49.832383 354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:49.832420 354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:49.834000 354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab
--principal=oryx/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:46849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=39813
--tserver_master_addrs=127.24.255.62:46751,127.24.255.61:38879,127.24.255.60:35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:49.834316 354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:49.834546 354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:49.839912 362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:49.839893 360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:49.839900 359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:49.840193 354 server_base.cc:1034] running on GCE node
I20260926 05:51:49.840378 354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:49.840648 354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:49.841823 354 hybrid_clock.cc:648] HybridClock initialized: now 1790401909841806 us; error 32 us; skew 500 ppm
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:49.844482 354 init.cc:377] Logged in from keytab as oryx/127.24.255.3@KRBTEST.COM (short username oryx)
I20260926 05:51:49.844918 354 webserver.cc:460] Webserver started at http://127.24.255.3:39813/ using document root <none> and password file <none>
I20260926 05:51:49.845100 354 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:49.845151 354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:49.846617 354 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:49.847218 369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.847461 354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:49.847546 354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "e0390f7fabb244bc902af3b9eab49ded"
format_stamp: "Formatted at 2026-09-26 05:51:48 on dist-test-slave-7fr8"
server_key: "b30a4aa20fe6528c598580e48183ee62"
server_key_iv: "7f8a142c2ee30d263a62d41dbd916f32"
server_key_version: "encryptionkey@0"
I20260926 05:51:49.847903 354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:49.855484 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.852524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35029 (local address 127.0.0.1:55380)
0926 05:51:49.852582 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.852594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.852656 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.852812 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.852813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.852815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:49.852864 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.852866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.853494 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.853495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.854650 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.854653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.855251 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.855252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.855310 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.855314 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.855356 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.855399 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 05:51:49.855577 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.852587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.24.255.60:35029)
0926 05:51:49.852643 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.852645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.852655 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:49.852719 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.852720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.852738 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.852793 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:49.852913 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.853429 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.854761 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.855173 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.855241 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.855293 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.855459 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.855511 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:51:49.856029 32509 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 05:51:49.856171 32538 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59ebed1574644d97a63c34f3b3107cef. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } } }
I20260926 05:51:49.856266 32538 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: This master's current role is: LEARNER
I20260926 05:51:49.860993 354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:49.863792 354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:49.863960 354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:49.864205 354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:49.864509 354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:49.864542 354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.864564 354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:49.864578 354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:49.870565 354 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:46849
I20260926 05:51:49.870643 482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:46849 every 8 connection(s)
I20260926 05:51:49.870960 354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:51:49.871498 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 354
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.61@KRBTEST.COM
I20260926 05:51:49.881472 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.872860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.24.255.60:35029)
0926 05:51:49.873464 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.873466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.873472 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:51:49.873494 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.873495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.873505 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.873558 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.874186 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.874914 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.876046 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.876349 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.876405 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.878251 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.878252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.878253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.878259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.879938 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.880307 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.880308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.880309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.880379 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.880566 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.880567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.880567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.880692 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.880779 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.881279 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.881355 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":46,"threads_started":1}
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
Sep 26 05:51:49 dist-test-slave-7fr8 krb5kdc[31671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401909, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.60@KRBTEST.COM
I20260926 05:51:49.884449 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.872675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.24.255.62:46751)
0926 05:51:49.873174 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.873175 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.873182 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:49.873205 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.873205 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.873215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.873276 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.874039 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.874988 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.876107 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.876422 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.876472 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.879321 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.879322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.879322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.879329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.881139 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.882654 (+ 1515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.882656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.882656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.883359 (+ 703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.883554 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.883555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.883556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.883743 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.883838 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.884262 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.884342 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":73,"threads_started":1}
I20260926 05:51:49.884581 494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.872633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.24.255.61:38879)
0926 05:51:49.872982 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.872984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.873095 (+ 111us) server_negotiation.cc:407] Connection header received
0926 05:51:49.873127 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.873128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.873142 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.873199 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.874032 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.874890 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.876030 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.876401 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.876451 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.878844 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.878846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.878846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.878852 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.881993 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.882398 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.882399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.882399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.882462 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.882694 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.882694 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.882695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.882828 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.882925 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.883592 (+ 667us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.884395 (+ 803us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":50,"threads_started":1}
I20260926 05:51:49.885967 488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.871831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38879 (local address 127.24.255.3:52951)
0926 05:51:49.872412 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.872419 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.873585 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.873961 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.873966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.874345 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.875833 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.875835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.877415 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.877416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.878265 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.878267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.878796 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.878797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.878953 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.879588 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.879599 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.882661 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.884276 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.884285 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.884288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.884586 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.884787 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.884788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.884788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.884921 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.885251 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.885257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.885449 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.885553 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.885725 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":88,"threads_started":1}
I20260926 05:51:49.886307 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.872066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:47801 (local address 127.24.255.62:46751)
0926 05:51:49.872111 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.872113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.872354 (+ 241us) server_negotiation.cc:407] Connection header received
0926 05:51:49.873654 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.873656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.873667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.873711 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.874962 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.875681 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.877991 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.878307 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.878360 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.882627 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.882629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.882629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.882637 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.884309 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.885332 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.885333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.885334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.885389 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.885580 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.885581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.885581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.885695 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.885751 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.886165 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.886233 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:51:49.886399 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.872090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:52951 (local address 127.24.255.61:38879)
0926 05:51:49.872206 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.872207 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.872486 (+ 279us) server_negotiation.cc:407] Connection header received
0926 05:51:49.873758 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.873760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.873769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.873811 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.875916 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.876579 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.878409 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.878677 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.878736 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.882741 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.882743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.882743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.882750 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.884201 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.884670 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.884671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.884672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.884725 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.884994 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.884995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.884995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.885115 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.885169 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.886221 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.886293 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 05:51:49.886456 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.871789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.3:47801)
0926 05:51:49.872283 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.872292 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.873449 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.874418 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.874419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.874577 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.874849 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.874855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.876686 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.876687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.877894 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.877897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.879134 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.879135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.879205 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.881003 (+ 1798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.881008 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.882524 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.885031 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.885032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.885033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.885233 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.885457 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.885458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.885459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.885516 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.885777 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.885777 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.885885 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.885948 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.886047 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":3,"spinlock_wait_cycles":8576,"thread_start_us":136,"threads_started":1}
I20260926 05:51:49.886729 485 heartbeater.cc:347] Connected to a master server at 127.24.255.61:38879
I20260926 05:51:49.886747 483 heartbeater.cc:347] Connected to a master server at 127.24.255.62:46751
I20260926 05:51:49.886837 485 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:49.886837 483 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:49.887063 485 heartbeater.cc:511] Master 127.24.255.61:38879 requested a full tablet report, sending...
I20260926 05:51:49.887060 483 heartbeater.cc:511] Master 127.24.255.62:46751 requested a full tablet report, sending...
I20260926 05:51:49.887439 31715 ts_manager.cc:194] Re-registered known tserver with Master: e0390f7fabb244bc902af3b9eab49ded (127.24.255.3:46849)
I20260926 05:51:49.887498 32313 ts_manager.cc:194] Registered new tserver with Master: e0390f7fabb244bc902af3b9eab49ded (127.24.255.3:46849)
I20260926 05:51:49.887773 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.872110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:38425 (local address 127.24.255.60:35029)
0926 05:51:49.872166 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.872169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.872322 (+ 153us) server_negotiation.cc:407] Connection header received
0926 05:51:49.878407 (+ 6085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.878409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.878425 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.878456 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:49.879862 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.880552 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.881679 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.881982 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.882023 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.884131 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:49.884133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:49.884133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:49.884141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:49.886213 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.886926 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.886927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.886927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.886984 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:49.887216 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:49.887217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:49.887217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:49.887340 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:49.887380 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.887611 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.887693 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 05:51:49.887936 31715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:47801
I20260926 05:51:49.886059 31716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 05:51:49.888290 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.871789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35029 (local address 127.24.255.3:38425)
0926 05:51:49.872182 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.872228 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.878332 (+ 6104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.878511 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.878512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.878939 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:49.879003 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.879005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.880645 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.880645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.881493 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.881495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.882153 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.882154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.882222 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.882686 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:49.882690 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:49.884036 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:49.886584 (+ 2548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.886585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.886585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.886778 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.887053 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:49.887053 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:49.887053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:49.887147 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:49.887422 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:49.887423 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:49.887459 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.887934 (+ 475us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.888046 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":81,"threads_started":1}
W20260926 05:51:49.888295 31716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:49.888614 484 heartbeater.cc:347] Connected to a master server at 127.24.255.60:35029
I20260926 05:51:49.888661 484 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:49.888721 484 heartbeater.cc:511] Master 127.24.255.60:35029 requested a full tablet report, sending...
I20260926 05:51:49.888967 32499 ts_manager.cc:194] Registered new tserver with Master: e0390f7fabb244bc902af3b9eab49ded (127.24.255.3:46849)
I20260926 05:51:49.888980 32471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: NON_VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: true } } }
I20260926 05:51:49.889057 32471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:49.890411 32274 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef: attempting to promote NON_VOTER ebe661be221845cd986e688e7b4f6041 to VOTER
I20260926 05:51:49.890532 32274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } }
I20260926 05:51:49.890877 32509 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 1 LEARNER]: Refusing update from remote peer 59ebed1574644d97a63c34f3b3107cef: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 05:51:49.890892 32323 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 FOLLOWER]: Refusing update from remote peer 59ebed1574644d97a63c34f3b3107cef: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 05:51:49.891065 32275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 05:51:49.891182 32275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 05:51:49.891736 32275 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ebe661be221845cd986e688e7b4f6041 (127.24.255.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } } }
I20260926 05:51:49.891883 32509 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ebe661be221845cd986e688e7b4f6041 (127.24.255.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } } }
I20260926 05:51:49.892695 32275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } } }
I20260926 05:51:49.892766 32275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:49.892771 32538 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } } }
I20260926 05:51:49.892832 32538 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:49.892814 32275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } } }
I20260926 05:51:49.892856 32275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:49.893165 32274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } } }
I20260926 05:51:49.893234 32274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:49.893326 32323 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ebe661be221845cd986e688e7b4f6041 (127.24.255.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } } }
I20260926 05:51:49.894394 32463 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59ebed1574644d97a63c34f3b3107cef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } } }
I20260926 05:51:49.894479 32463 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:49.898272 511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.894743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.24.255.3:46849)
0926 05:51:49.894921 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.894926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.894935 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:49.894970 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.894972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.894986 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.895219 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:49.895274 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.896003 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.897172 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.897914 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.897976 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.898056 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.898155 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.898193 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
I20260926 05:51:49.898262 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.893765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:46849 (local address 127.0.0.1:39650)
0926 05:51:49.894027 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.894036 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.894062 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.895096 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.895097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.895099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:49.895150 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.895152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.896102 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.896103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.897089 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.897092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.897993 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.897994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.898056 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.898059 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.898135 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.898179 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":85,"threads_started":1}
I20260926 05:51:49.898581 509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.893932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:37079 (local address 127.0.0.1:44886)
0926 05:51:49.894054 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.894065 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.894150 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.894966 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.894966 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.894968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:49.895005 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.895007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.896401 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.896402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.897301 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.897304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.898324 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.898325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.898390 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.898395 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.898446 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.898490 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":48,"thread_start_us":65,"threads_started":1}
I20260926 05:51:49.898713 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.893713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:39449 (local address 127.0.0.1:48090)
0926 05:51:49.893897 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.893908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.893955 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.894483 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.894484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.894487 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:49.894535 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.894537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.896411 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.896412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.897572 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.897575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.898482 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.898482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.898534 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.898537 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.898564 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.898596 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":24}
I20260926 05:51:49.898829 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.894592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.24.255.2:37079)
0926 05:51:49.894750 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.894755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.894767 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:51:49.894806 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.894808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.894826 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.894906 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:49.895077 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.896298 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.897396 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.898156 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.898221 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.898290 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.898701 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.898752 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260926 05:51:49.898849 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.893844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.24.255.1:39449)
0926 05:51:49.894325 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.894330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.894340 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:49.894381 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.894383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.894401 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.894476 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:49.894584 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.896328 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.897665 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.898400 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.898636 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.898720 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.898742 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.898779 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":88,"threads_started":1}
I20260926 05:51:49.899116 417 tablet_service.cc:1467] Processing CreateTablet for tablet 26a75450d2cf4fba9e57da1cc1648f11 (DEFAULT_TABLE table=test-table [id=583d82505b3843cf9d35b2a09b75837f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:49.899391 32612 tablet_service.cc:1467] Processing CreateTablet for tablet 26a75450d2cf4fba9e57da1cc1648f11 (DEFAULT_TABLE table=test-table [id=583d82505b3843cf9d35b2a09b75837f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:49.899487 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a75450d2cf4fba9e57da1cc1648f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:49.899665 32612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a75450d2cf4fba9e57da1cc1648f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:49.902590 513 tablet_bootstrap.cc:492] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58: Bootstrap starting.
I20260926 05:51:49.902961 32749 tablet_service.cc:1467] Processing CreateTablet for tablet 26a75450d2cf4fba9e57da1cc1648f11 (DEFAULT_TABLE table=test-table [id=583d82505b3843cf9d35b2a09b75837f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:49.903326 32749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a75450d2cf4fba9e57da1cc1648f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:49.903648 513 tablet_bootstrap.cc:654] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:49.903901 513 log.cc:824] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:49.904261 512 tablet_bootstrap.cc:492] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded: Bootstrap starting.
I20260926 05:51:49.904770 513 tablet_bootstrap.cc:492] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58: No bootstrap required, opened a new log
I20260926 05:51:49.904842 513 ts_tablet_manager.cc:1397] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:51:49.904986 512 tablet_bootstrap.cc:654] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:49.905278 512 log.cc:824] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:49.906222 512 tablet_bootstrap.cc:492] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded: No bootstrap required, opened a new log
I20260926 05:51:49.906301 512 ts_tablet_manager.cc:1397] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:51:49.906603 516 tablet_bootstrap.cc:492] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35: Bootstrap starting.
I20260926 05:51:49.906955 513 raft_consensus.cc:348] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } }
I20260926 05:51:49.907157 513 raft_consensus.cc:374] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:49.907195 513 raft_consensus.cc:729] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c39876dc73428d87aadbba9ed9de58, State: Initialized, Role: FOLLOWER
I20260926 05:51:49.907267 516 tablet_bootstrap.cc:654] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:49.907366 513 consensus_queue.cc:260] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } }
I20260926 05:51:49.907546 516 log.cc:824] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:49.907658 513 ts_tablet_manager.cc:1428] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:51:49.907721 32678 heartbeater.cc:503] Master 127.24.255.62:46751 was elected leader, sending a full tablet report...
I20260926 05:51:49.909041 512 raft_consensus.cc:348] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } }
I20260926 05:51:49.909226 512 raft_consensus.cc:374] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:49.909264 512 raft_consensus.cc:729] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0390f7fabb244bc902af3b9eab49ded, State: Initialized, Role: FOLLOWER
I20260926 05:51:49.909266 516 tablet_bootstrap.cc:492] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35: No bootstrap required, opened a new log
I20260926 05:51:49.909391 512 consensus_queue.cc:260] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } }
I20260926 05:51:49.909478 516 ts_tablet_manager.cc:1397] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:51:49.909720 483 heartbeater.cc:503] Master 127.24.255.62:46751 was elected leader, sending a full tablet report...
I20260926 05:51:49.909694 512 ts_tablet_manager.cc:1428] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:51:49.911949 516 raft_consensus.cc:348] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } }
I20260926 05:51:49.912154 516 raft_consensus.cc:374] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:49.912192 516 raft_consensus.cc:729] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0513b0a4ecb94eb1935b1eda3d179c35, State: Initialized, Role: FOLLOWER
I20260926 05:51:49.912299 516 consensus_queue.cc:260] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } }
I20260926 05:51:49.912568 516 ts_tablet_manager.cc:1428] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:51:49.912634 347 heartbeater.cc:503] Master 127.24.255.62:46751 was elected leader, sending a full tablet report...
I20260926 05:51:49.964340 520 raft_consensus.cc:482] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:49.964532 520 raft_consensus.cc:504] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } }
I20260926 05:51:49.964917 520 leader_election.cc:288] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1c39876dc73428d87aadbba9ed9de58 (127.24.255.1:39449), e0390f7fabb244bc902af3b9eab49ded (127.24.255.3:46849)
I20260926 05:51:49.968860 511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.965367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:50783 (local address 127.24.255.3:46849)
0926 05:51:49.965458 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.965460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.965468 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:49.965497 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.965498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.965507 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.965561 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:49.965751 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.966497 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.967914 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.968483 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.968568 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.968580 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.968729 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.968779 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 05:51:49.968868 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.965241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:46849 (local address 127.24.255.2:50783)
0926 05:51:49.965344 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.965353 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.965393 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.965600 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.965601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.965603 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:49.965679 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.965681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.966596 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.966598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.967797 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.967801 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.968584 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.968586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.968666 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.968672 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.968722 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.968789 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 05:51:49.968972 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.965108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:39449 (local address 127.24.255.2:40425)
0926 05:51:49.965176 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:49.965188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:49.965231 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:49.965761 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:49.965762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:49.965763 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:49.965805 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.965806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.966789 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.966790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.967994 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:49.967997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.968741 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.968742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:49.968815 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.968817 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:51:49.968859 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 05:51:49.968909 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 05:51:49.969545 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.965479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:40425 (local address 127.24.255.1:39449)
0926 05:51:49.965520 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.965522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.965529 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:49.965550 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.965551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.965558 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.965601 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:49.965879 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.966709 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.968100 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.968640 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.968703 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.968712 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.969433 (+ 721us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.969481 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:51:49.969668 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a75450d2cf4fba9e57da1cc1648f11" candidate_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0390f7fabb244bc902af3b9eab49ded" is_pre_election: true
I20260926 05:51:49.969834 437 raft_consensus.cc:2454] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0513b0a4ecb94eb1935b1eda3d179c35 in term 0.
I20260926 05:51:49.969990 32632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a75450d2cf4fba9e57da1cc1648f11" candidate_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c39876dc73428d87aadbba9ed9de58" is_pre_election: true
I20260926 05:51:49.970130 32632 raft_consensus.cc:2454] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0513b0a4ecb94eb1935b1eda3d179c35 in term 0.
I20260926 05:51:49.970140 32705 leader_election.cc:302] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0513b0a4ecb94eb1935b1eda3d179c35, e0390f7fabb244bc902af3b9eab49ded; no voters:
I20260926 05:51:49.970379 520 raft_consensus.cc:2781] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:49.970446 520 raft_consensus.cc:482] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:49.970474 520 raft_consensus.cc:3037] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:49.971233 520 raft_consensus.cc:504] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } }
I20260926 05:51:49.971365 520 leader_election.cc:288] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [CANDIDATE]: Term 1 election: Requested vote from peers e1c39876dc73428d87aadbba9ed9de58 (127.24.255.1:39449), e0390f7fabb244bc902af3b9eab49ded (127.24.255.3:46849)
I20260926 05:51:49.971506 32632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a75450d2cf4fba9e57da1cc1648f11" candidate_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c39876dc73428d87aadbba9ed9de58"
I20260926 05:51:49.971585 32632 raft_consensus.cc:3037] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:49.972852 32632 raft_consensus.cc:2454] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0513b0a4ecb94eb1935b1eda3d179c35 in term 1.
I20260926 05:51:49.973044 32702 leader_election.cc:302] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0513b0a4ecb94eb1935b1eda3d179c35, e1c39876dc73428d87aadbba9ed9de58; no voters:
I20260926 05:51:49.973184 520 raft_consensus.cc:2781] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:49.973336 520 raft_consensus.cc:686] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [term 1 LEADER]: Becoming Leader. State: Replica: 0513b0a4ecb94eb1935b1eda3d179c35, State: Running, Role: LEADER
I20260926 05:51:49.973320 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a75450d2cf4fba9e57da1cc1648f11" candidate_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0390f7fabb244bc902af3b9eab49ded"
I20260926 05:51:49.973409 437 raft_consensus.cc:3037] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:49.973567 520 consensus_queue.cc:237] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } }
I20260926 05:51:49.974336 437 raft_consensus.cc:2454] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0513b0a4ecb94eb1935b1eda3d179c35 in term 1.
I20260926 05:51:49.974778 31716 catalog_manager.cc:5495] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0513b0a4ecb94eb1935b1eda3d179c35 (127.24.255.2). New cstate: current_term: 1 leader_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } health_report { overall_health: HEALTHY } } }
W20260926 05:51:49.985188 350 tablet.cc:2369] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:51:49.990455 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:49.987690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.24.255.2:37079)
0926 05:51:49.987756 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:49.987758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:49.987767 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:49.987804 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:49.987805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:49.987821 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:49.987896 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:49.988128 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.988672 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.989521 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:49.989714 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:49.989777 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:49.990039 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:49.990213 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:49.990295 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:49.990344 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 05:51:49.990371 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:51:49.992281 437 raft_consensus.cc:1261] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded [term 1 FOLLOWER]: Refusing update from remote peer 0513b0a4ecb94eb1935b1eda3d179c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:49.992573 32632 raft_consensus.cc:1261] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58 [term 1 FOLLOWER]: Refusing update from remote peer 0513b0a4ecb94eb1935b1eda3d179c35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:49.992659 520 consensus_queue.cc:1035] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:49.992810 520 consensus_queue.cc:1035] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:49.995918 524 mvcc.cc:205] Tried to move back new op lower bound from 7333486223325597696 to 7333486223252389888. Current Snapshot: MvccSnapshot[applied={T|T < 7333486223325597696}]
I20260926 05:51:49.998421 31715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:49.998533 31715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:49.999929 31715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59ebed1574644d97a63c34f3b3107cef: Sending DeleteTablet for 3 replicas of tablet 26a75450d2cf4fba9e57da1cc1648f11
I20260926 05:51:50.000226 32612 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a75450d2cf4fba9e57da1cc1648f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:49 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:48090
I20260926 05:51:50.000236 417 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a75450d2cf4fba9e57da1cc1648f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:49 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:39650
I20260926 05:51:50.001468 534 tablet_replica.cc:330] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded: stopping tablet replica
I20260926 05:51:50.001609 534 raft_consensus.cc:2229] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:50.001643 32749 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a75450d2cf4fba9e57da1cc1648f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:49 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:44886
I20260926 05:51:50.001729 534 raft_consensus.cc:2258] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:50.001897 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 31684
I20260926 05:51:50.002486 534 ts_tablet_manager.cc:1905] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:50.004499 534 ts_tablet_manager.cc:1918] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:50.004601 534 log.cc:1196] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/26a75450d2cf4fba9e57da1cc1648f11
I20260926 05:51:50.004875 534 ts_tablet_manager.cc:1939] T 26a75450d2cf4fba9e57da1cc1648f11 P e0390f7fabb244bc902af3b9eab49ded: Deleting consensus metadata
I20260926 05:51:50.006358 536 tablet_replica.cc:330] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35: stopping tablet replica
I20260926 05:51:50.006534 536 raft_consensus.cc:2229] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:51:50.006695 536 raft_consensus.cc:2258] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:50.007220 536 ts_tablet_manager.cc:1905] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:50.009137 536 ts_tablet_manager.cc:1918] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:50.009267 536 log.cc:1196] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/26a75450d2cf4fba9e57da1cc1648f11
I20260926 05:51:50.009639 536 ts_tablet_manager.cc:1939] T 26a75450d2cf4fba9e57da1cc1648f11 P 0513b0a4ecb94eb1935b1eda3d179c35: Deleting consensus metadata
W20260926 05:51:50.010197 32702 connection.cc:535] server connection from 127.0.0.1:44886 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 05:51:50.010281 373 connection.cc:535] server connection from 127.0.0.1:39650 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 05:51:50.011185 533 tablet_replica.cc:330] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58: stopping tablet replica
I20260926 05:51:50.011353 533 raft_consensus.cc:2229] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:50.011528 533 raft_consensus.cc:2258] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:50.013553 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:50.000930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.24.255.60:35029)
0926 05:51:50.000974 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:50.000976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:50.001919 (+ 943us) server_negotiation.cc:407] Connection header received
0926 05:51:50.002014 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:50.002015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:50.002025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:50.002053 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:50.003979 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:50.004730 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:50.006444 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:50.006744 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:50.006794 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:50.009564 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:50.009566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:50.009567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:50.009577 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:50.011994 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:50.012477 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:50.012478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:50.012479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:50.012547 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:50.012786 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:50.012786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:50.012787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:50.012939 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:50.013044 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:50.013387 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 05:51:50.013458 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:51:50.013549 533 ts_tablet_manager.cc:1905] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:50.015714 533 ts_tablet_manager.cc:1918] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:50.015807 533 log.cc:1196] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-0/wal/wals/26a75450d2cf4fba9e57da1cc1648f11
I20260926 05:51:50.016070 533 ts_tablet_manager.cc:1939] T 26a75450d2cf4fba9e57da1cc1648f11 P e1c39876dc73428d87aadbba9ed9de58: Deleting consensus metadata
W20260926 05:51:50.016574 32566 connection.cc:441] server connection from 127.0.0.1:48090 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48090 (request call id 1) could send its response
I20260926 05:51:50.021404 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:50.012867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.24.255.60:35029)
0926 05:51:50.012909 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:50.012911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:50.012918 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:50.013009 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:50.013010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:50.013026 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:50.013058 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:50.013944 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:50.014772 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:50.016002 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:50.016336 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:50.016389 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:50.018144 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:50.018145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:50.018146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:50.018153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:50.019643 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:50.020026 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:50.020027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:50.020027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:50.020103 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:50.020303 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:50.020304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:50.020304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:50.020665 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:50.020773 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:50.021208 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 05:51:50.021289 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:51:50.022014 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:50.013095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58534 (local address 127.24.255.61:38879)
0926 05:51:50.013152 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:50.013154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:50.014978 (+ 1824us) server_negotiation.cc:407] Connection header received
0926 05:51:50.015106 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:50.015107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:50.015119 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:50.015145 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:50.015892 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:50.016640 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:50.017508 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:50.017701 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:50.017756 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:50.019140 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:50.019141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:50.019141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:50.019149 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:50.020668 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:50.021063 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:50.021064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:50.021064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:50.021123 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:50.021328 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:50.021329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:50.021329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:50.021463 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:50.021529 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:50.021838 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 05:51:50.021918 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:51:50.997784 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:50.997205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.3:60909)
0926 05:51:50.997633 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:50.997725 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":146,"threads_started":1}
W20260926 05:51:50.998064 483 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:46751 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
I20260926 05:51:50.998073 544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:50.997615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.2:40723)
0926 05:51:50.997846 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:50.998027 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":40,"threads_started":1}
W20260926 05:51:50.998328 347 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:46751 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
I20260926 05:51:50.998880 545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:50.998664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.1:46387)
0926 05:51:50.998765 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:50.998826 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
W20260926 05:51:50.999053 32678 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:46751 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
I20260926 05:51:51.501184 546 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59ebed1574644d97a63c34f3b3107cef)
I20260926 05:51:51.501330 546 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } }
I20260926 05:51:51.501605 546 leader_election.cc:288] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59ebed1574644d97a63c34f3b3107cef (127.24.255.62:46751), d20eeed10ddf459cbcffe8ec5d9579f4 (127.24.255.61:38879)
I20260926 05:51:51.501700 547 raft_consensus.cc:482] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59ebed1574644d97a63c34f3b3107cef)
I20260926 05:51:51.501857 547 raft_consensus.cc:504] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } }
I20260926 05:51:51.501962 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.501763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:57956)
0926 05:51:51.501869 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.501921 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":68,"threads_started":1}
W20260926 05:51:51.502074 32484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
I20260926 05:51:51.502311 547 leader_election.cc:288] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59ebed1574644d97a63c34f3b3107cef (127.24.255.62:46751), ebe661be221845cd986e688e7b4f6041 (127.24.255.60:35029)
I20260926 05:51:51.502560 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.502401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:57986)
0926 05:51:51.502505 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.502532 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 05:51:51.502631 32484 leader_election.cc:334] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 59ebed1574644d97a63c34f3b3107cef (127.24.255.62:46751): Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
I20260926 05:51:51.502727 553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.502371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:57970)
0926 05:51:51.502541 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.502586 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":46,"threads_started":1}
W20260926 05:51:51.502821 32298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
I20260926 05:51:51.503275 553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.503171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:58000)
0926 05:51:51.503219 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.503248 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 05:51:51.503402 32298 leader_election.cc:334] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 59ebed1574644d97a63c34f3b3107cef (127.24.255.62:46751): Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
I20260926 05:51:51.505119 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.501771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38879 (local address 127.0.0.1:58536)
0926 05:51:51.501966 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.502008 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:51.502076 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:51.502264 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:51.502265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:51.502267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:51.502539 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.502543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.503451 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.503452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.504282 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.504285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.504886 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.504887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.504949 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.504954 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:51.505006 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 05:51:51.505057 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":48,"threads_started":1}
I20260926 05:51:51.505139 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.501874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.24.255.61:38879)
0926 05:51:51.501987 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:51.501990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:51.502064 (+ 74us) server_negotiation.cc:407] Connection header received
0926 05:51:51.502165 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:51.502166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:51.502178 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:51.502234 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:51.502627 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.503390 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.504361 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.504813 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.504875 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.504889 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:51.505028 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 05:51:51.505073 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
I20260926 05:51:51.505345 32323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ebe661be221845cd986e688e7b4f6041" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" is_pre_election: true
I20260926 05:51:51.505453 32323 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebe661be221845cd986e688e7b4f6041 in term 1.
I20260926 05:51:51.505594 32486 leader_election.cc:302] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d20eeed10ddf459cbcffe8ec5d9579f4, ebe661be221845cd986e688e7b4f6041; no voters: 59ebed1574644d97a63c34f3b3107cef
I20260926 05:51:51.505614 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.502400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35029 (local address 127.0.0.1:55406)
0926 05:51:51.502741 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.502751 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:51.502810 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:51.503006 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:51.503007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:51.503009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:51.503049 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.503051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.503799 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.503800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.504652 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.504655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.505360 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.505361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.505438 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.505443 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:51.505504 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:51:51.505549 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":2,"thread_start_us":37,"threads_started":1}
I20260926 05:51:51.505666 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.502592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.24.255.60:35029)
0926 05:51:51.502712 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:51.502714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:51.502861 (+ 147us) server_negotiation.cc:407] Connection header received
0926 05:51:51.502883 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:51.502883 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:51.502893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:51.502970 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:51.503113 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.503722 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.504724 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.505167 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.505331 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.505344 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:51.505567 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 05:51:51.505612 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260926 05:51:51.505697 546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 05:51:51.505725 546 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59ebed1574644d97a63c34f3b3107cef)
I20260926 05:51:51.505743 546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:51:51.505929 32509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ebe661be221845cd986e688e7b4f6041" is_pre_election: true
I20260926 05:51:51.506542 546 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } }
I20260926 05:51:51.506675 546 leader_election.cc:288] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [CANDIDATE]: Term 2 election: Requested vote from peers 59ebed1574644d97a63c34f3b3107cef (127.24.255.62:46751), d20eeed10ddf459cbcffe8ec5d9579f4 (127.24.255.61:38879)
I20260926 05:51:51.506700 32509 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d20eeed10ddf459cbcffe8ec5d9579f4 in current term 2: Already voted for candidate ebe661be221845cd986e688e7b4f6041 in this term.
I20260926 05:51:51.506930 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.506782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:58014)
0926 05:51:51.506877 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.506906 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 05:51:51.506916 32297 leader_election.cc:302] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20eeed10ddf459cbcffe8ec5d9579f4; no voters: 59ebed1574644d97a63c34f3b3107cef, ebe661be221845cd986e688e7b4f6041
I20260926 05:51:51.507062 32323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ebe661be221845cd986e688e7b4f6041" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4"
I20260926 05:51:51.507119 32323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:51:51.507467 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.507290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:58018)
0926 05:51:51.507413 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.507444 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260926 05:51:51.507572 32484 leader_election.cc:334] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 59ebed1574644d97a63c34f3b3107cef (127.24.255.62:46751): Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
I20260926 05:51:51.507853 32323 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebe661be221845cd986e688e7b4f6041 in term 2.
I20260926 05:51:51.508047 547 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:51:51.508051 32486 leader_election.cc:302] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d20eeed10ddf459cbcffe8ec5d9579f4, ebe661be221845cd986e688e7b4f6041; no voters: 59ebed1574644d97a63c34f3b3107cef
I20260926 05:51:51.508157 546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:51:51.508329 546 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [term 2 LEADER]: Becoming Leader. State: Replica: ebe661be221845cd986e688e7b4f6041, State: Running, Role: LEADER
I20260926 05:51:51.508420 546 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } }
I20260926 05:51:51.508831 556 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe661be221845cd986e688e7b4f6041. Latest consensus state: current_term: 2 leader_uuid: "ebe661be221845cd986e688e7b4f6041" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } } }
I20260926 05:51:51.508991 556 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:51.509325 558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:51.509922 558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:51.510017 558 catalog_manager.cc:1259] Loaded cluster ID: 76801b8588544dd9add52d55f500263a
I20260926 05:51:51.510057 558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:51.510416 558 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:51.510504 558 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: Loaded TSK: 0
I20260926 05:51:51.510641 558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:51.571695 32499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:51:51.573300 32499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:51.574349 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.574231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:58052)
0926 05:51:51.574282 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.574318 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 05:51:51.574401 32323 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [term 2 FOLLOWER]: Refusing update from remote peer ebe661be221845cd986e688e7b4f6041: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 05:51:51.574659 556 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 05:51:51.574749 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.574652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:58062)
0926 05:51:51.574697 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.574723 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:51:51.574949 32484 consensus_peers.cc:487] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 -> Peer 59ebed1574644d97a63c34f3b3107cef (127.24.255.62:46751): Couldn't send request to peer 59ebed1574644d97a63c34f3b3107cef. Status: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 05:51:51.575244 547 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe661be221845cd986e688e7b4f6041. Latest consensus state: current_term: 2 leader_uuid: "ebe661be221845cd986e688e7b4f6041" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } } }
I20260926 05:51:51.575341 547 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:51.575676 559 mvcc.cc:205] Tried to move back new op lower bound from 7333486229806370816 to 7333486229539078144. Current Snapshot: MvccSnapshot[applied={T|T < 7333486229806370816}]
I20260926 05:51:51.575791 556 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ebe661be221845cd986e688e7b4f6041" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } } }
I20260926 05:51:51.575866 556 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:51.575968 547 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ebe661be221845cd986e688e7b4f6041" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "59ebed1574644d97a63c34f3b3107cef" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46751 } } peers { permanent_uuid: "d20eeed10ddf459cbcffe8ec5d9579f4" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38879 } attrs { promote: false } } peers { permanent_uuid: "ebe661be221845cd986e688e7b4f6041" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35029 } attrs { promote: false } } }
I20260926 05:51:51.576063 547 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20eeed10ddf459cbcffe8ec5d9579f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:51.581645 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.578674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.24.255.2:37079)
0926 05:51:51.578947 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:51.578949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:51.578955 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:51:51.578978 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:51.578979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:51.578988 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:51.579041 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:51.579122 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.579849 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.580733 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.581291 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.581362 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.581374 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:51.581525 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 05:51:51.581566 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":60,"threads_started":1}
I20260926 05:51:51.581646 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.578408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:39449 (local address 127.0.0.1:48100)
0926 05:51:51.578600 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.578607 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:51.578639 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:51.578939 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:51.578939 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:51.578941 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:51.578971 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.578972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.579636 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.579637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.580764 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.580767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.581401 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.581401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.581451 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.581456 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:51.581506 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 05:51:51.581548 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":30}
I20260926 05:51:51.581648 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.578395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:37079 (local address 127.0.0.1:44910)
0926 05:51:51.578453 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.578463 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:51.578516 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:51.579050 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:51.579050 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:51.579052 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:51.579078 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.579078 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.579905 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.579906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.580643 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.580645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.581384 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.581385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.581451 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.581456 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:51.581505 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 05:51:51.581548 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 05:51:51.581957 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.578500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:46849 (local address 127.0.0.1:39666)
0926 05:51:51.578652 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.578663 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:51.578701 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:51.578880 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:51.578881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:51.578882 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:51.578931 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.578933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.579846 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.579846 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.580954 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.580956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.581667 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.581667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.581799 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.581804 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:51.581856 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 05:51:51.581897 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
I20260926 05:51:51.581964 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.578591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.24.255.3:46849)
0926 05:51:51.578781 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:51.578783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:51.578790 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:51.578812 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:51.578813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:51.578822 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:51.578877 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:51.578988 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.579767 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.581029 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.581564 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.581625 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.581637 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:51.581875 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 05:51:51.581913 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":46,"threads_started":1}
I20260926 05:51:51.582422 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.578472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.24.255.1:39449)
0926 05:51:51.578739 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:51.578740 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:51.578748 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:51.578773 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:51.578773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:51.578784 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:51.578875 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:51.579045 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.579561 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.580847 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.581298 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.581359 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.581374 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:51.582328 (+ 954us) server_negotiation.cc:299] Negotiation successful
0926 05:51:51.582370 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":69,"threads_started":1}
I20260926 05:51:51.582435 417 tablet_service.cc:1467] Processing CreateTablet for tablet 035a8860d2c446acbdfe320a70bcad0f (DEFAULT_TABLE table=test-table [id=b2b3da15175240d9adc03f42b2a16614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:51.582580 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035a8860d2c446acbdfe320a70bcad0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:51.582620 32612 tablet_service.cc:1467] Processing CreateTablet for tablet 035a8860d2c446acbdfe320a70bcad0f (DEFAULT_TABLE table=test-table [id=b2b3da15175240d9adc03f42b2a16614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:51.582623 32749 tablet_service.cc:1467] Processing CreateTablet for tablet 035a8860d2c446acbdfe320a70bcad0f (DEFAULT_TABLE table=test-table [id=b2b3da15175240d9adc03f42b2a16614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:51.582729 32612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035a8860d2c446acbdfe320a70bcad0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:51.582729 32749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 035a8860d2c446acbdfe320a70bcad0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:51.584107 569 tablet_bootstrap.cc:492] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58: Bootstrap starting.
I20260926 05:51:51.584226 568 tablet_bootstrap.cc:492] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded: Bootstrap starting.
I20260926 05:51:51.584530 569 tablet_bootstrap.cc:654] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:51.584661 568 tablet_bootstrap.cc:654] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:51.584811 570 tablet_bootstrap.cc:492] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35: Bootstrap starting.
I20260926 05:51:51.585139 570 tablet_bootstrap.cc:654] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:51.585175 568 tablet_bootstrap.cc:492] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded: No bootstrap required, opened a new log
I20260926 05:51:51.585217 568 ts_tablet_manager.cc:1397] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:51.585407 569 tablet_bootstrap.cc:492] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58: No bootstrap required, opened a new log
I20260926 05:51:51.585366 568 raft_consensus.cc:348] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } }
I20260926 05:51:51.585448 568 raft_consensus.cc:374] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:51.585453 569 ts_tablet_manager.cc:1397] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:51.585470 568 raft_consensus.cc:729] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0390f7fabb244bc902af3b9eab49ded, State: Initialized, Role: FOLLOWER
I20260926 05:51:51.585530 568 consensus_queue.cc:260] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } }
I20260926 05:51:51.585561 569 raft_consensus.cc:348] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } }
I20260926 05:51:51.585644 569 raft_consensus.cc:374] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:51.585671 569 raft_consensus.cc:729] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c39876dc73428d87aadbba9ed9de58, State: Initialized, Role: FOLLOWER
I20260926 05:51:51.585687 568 ts_tablet_manager.cc:1428] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:51:51.585736 569 consensus_queue.cc:260] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } }
I20260926 05:51:51.585893 569 ts_tablet_manager.cc:1428] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:51.586089 570 tablet_bootstrap.cc:492] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35: No bootstrap required, opened a new log
I20260926 05:51:51.586162 570 ts_tablet_manager.cc:1397] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:51.586371 570 raft_consensus.cc:348] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } }
I20260926 05:51:51.586447 570 raft_consensus.cc:374] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:51.586467 570 raft_consensus.cc:729] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0513b0a4ecb94eb1935b1eda3d179c35, State: Initialized, Role: FOLLOWER
I20260926 05:51:51.586519 570 consensus_queue.cc:260] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } }
I20260926 05:51:51.586721 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.586267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.3:46335)
0926 05:51:51.586639 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.586678 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":76,"threads_started":1}
I20260926 05:51:51.586720 570 ts_tablet_manager.cc:1428] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 05:51:51.586797 32679 heartbeater.cc:503] Master 127.24.255.60:35029 was elected leader, sending a full tablet report...
I20260926 05:51:51.586951 577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.586426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.1:33955)
0926 05:51:51.586891 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.586924 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":64,"threads_started":1}
I20260926 05:51:51.587113 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.586910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.2:58721)
0926 05:51:51.587046 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.587080 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260926 05:51:51.586987 484 heartbeater.cc:503] Master 127.24.255.60:35029 was elected leader, sending a full tablet report...
I20260926 05:51:51.587495 348 heartbeater.cc:503] Master 127.24.255.60:35029 was elected leader, sending a full tablet report...
I20260926 05:51:51.587697 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.587573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.2:36171)
0926 05:51:51.587600 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.587670 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 05:51:51.591699 32681 tablet.cc:2369] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:51:51.622974 486 tablet.cc:2369] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:51:51.862550 574 raft_consensus.cc:482] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:51.862658 574 raft_consensus.cc:504] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } }
I20260926 05:51:51.863008 574 leader_election.cc:288] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0513b0a4ecb94eb1935b1eda3d179c35 (127.24.255.2:37079), e1c39876dc73428d87aadbba9ed9de58 (127.24.255.1:39449)
I20260926 05:51:51.866254 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.863172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:39449 (local address 127.24.255.3:46769)
0926 05:51:51.863419 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.863430 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:51.863489 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:51.863628 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:51.863629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:51.863631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:51.863697 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.863698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.864299 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.864300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.865137 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.865140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.865848 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.865850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.865955 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.865963 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 05:51:51.866029 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 05:51:51.866091 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":139,"thread_start_us":101,"threads_started":1}
I20260926 05:51:51.866443 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.863299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:46769 (local address 127.24.255.1:39449)
0926 05:51:51.863348 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:51.863350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:51.863514 (+ 164us) server_negotiation.cc:407] Connection header received
0926 05:51:51.863543 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:51.863544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:51.863553 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:51.863609 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:51.863768 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.864215 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.865238 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.865619 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.865679 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.865688 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:51.866172 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 05:51:51.866366 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:51:51.866710 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.863165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:37079 (local address 127.24.255.3:33727)
0926 05:51:51.863227 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.863238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:51.863287 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:51.863542 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:51.863543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:51.863546 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:51.863628 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.863630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.864293 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.864294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.865132 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:51.865135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.866330 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.866334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:51.866421 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.866431 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 05:51:51.866578 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 05:51:51.866635 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":35}
I20260926 05:51:51.866760 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.863362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:33727 (local address 127.24.255.2:37079)
0926 05:51:51.863403 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:51.863405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:51.863412 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:51.863438 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:51.863439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:51.863449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:51.863516 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:51.863705 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.864202 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.865239 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.865837 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.865973 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.866002 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:51.866633 (+ 631us) server_negotiation.cc:299] Negotiation successful
0926 05:51:51.866702 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:51:51.866743 32632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035a8860d2c446acbdfe320a70bcad0f" candidate_uuid: "e0390f7fabb244bc902af3b9eab49ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c39876dc73428d87aadbba9ed9de58" is_pre_election: true
I20260926 05:51:51.866885 32632 raft_consensus.cc:2454] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0390f7fabb244bc902af3b9eab49ded in term 0.
I20260926 05:51:51.866995 301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035a8860d2c446acbdfe320a70bcad0f" candidate_uuid: "e0390f7fabb244bc902af3b9eab49ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" is_pre_election: true
I20260926 05:51:51.867147 301 raft_consensus.cc:2454] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0390f7fabb244bc902af3b9eab49ded in term 0.
I20260926 05:51:51.867146 370 leader_election.cc:302] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0390f7fabb244bc902af3b9eab49ded, e1c39876dc73428d87aadbba9ed9de58; no voters:
I20260926 05:51:51.867416 574 raft_consensus.cc:2781] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:51.867477 574 raft_consensus.cc:482] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:51.867539 574 raft_consensus.cc:3037] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:51.868237 574 raft_consensus.cc:504] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } }
I20260926 05:51:51.868363 574 leader_election.cc:288] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [CANDIDATE]: Term 1 election: Requested vote from peers 0513b0a4ecb94eb1935b1eda3d179c35 (127.24.255.2:37079), e1c39876dc73428d87aadbba9ed9de58 (127.24.255.1:39449)
I20260926 05:51:51.868537 301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035a8860d2c446acbdfe320a70bcad0f" candidate_uuid: "e0390f7fabb244bc902af3b9eab49ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0513b0a4ecb94eb1935b1eda3d179c35"
I20260926 05:51:51.868614 301 raft_consensus.cc:3037] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:51.868619 32632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "035a8860d2c446acbdfe320a70bcad0f" candidate_uuid: "e0390f7fabb244bc902af3b9eab49ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c39876dc73428d87aadbba9ed9de58"
I20260926 05:51:51.868711 32632 raft_consensus.cc:3037] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:51.869443 301 raft_consensus.cc:2454] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0390f7fabb244bc902af3b9eab49ded in term 1.
I20260926 05:51:51.869443 32632 raft_consensus.cc:2454] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0390f7fabb244bc902af3b9eab49ded in term 1.
I20260926 05:51:51.869673 373 leader_election.cc:302] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0513b0a4ecb94eb1935b1eda3d179c35, e0390f7fabb244bc902af3b9eab49ded; no voters:
I20260926 05:51:51.869787 574 raft_consensus.cc:2781] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:51.869992 574 raft_consensus.cc:686] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [term 1 LEADER]: Becoming Leader. State: Replica: e0390f7fabb244bc902af3b9eab49ded, State: Running, Role: LEADER
I20260926 05:51:51.870208 574 consensus_queue.cc:237] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } } peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } }
I20260926 05:51:51.870543 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.870364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.3:35229)
0926 05:51:51.870453 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.870512 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 05:51:51.871064 32498 catalog_manager.cc:5495] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded reported cstate change: term changed from 0 to 1, leader changed from <none> to e0390f7fabb244bc902af3b9eab49ded (127.24.255.3). New cstate: current_term: 1 leader_uuid: "e0390f7fabb244bc902af3b9eab49ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0390f7fabb244bc902af3b9eab49ded" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 46849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 } health_report { overall_health: UNKNOWN } } }
I20260926 05:51:51.939189 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.936288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39672 (local address 127.24.255.3:46849)
0926 05:51:51.936373 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:51.936374 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:51.936383 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:51.936424 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:51.936425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:51.936448 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:51.936537 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:51.936747 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.937241 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.938135 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:51.938382 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:51.938445 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:51.938772 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:51.938895 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:51.938974 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:51.939103 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 05:51:51.939125 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 05:51:51.940725 32632 raft_consensus.cc:1261] T 035a8860d2c446acbdfe320a70bcad0f P e1c39876dc73428d87aadbba9ed9de58 [term 1 FOLLOWER]: Refusing update from remote peer e0390f7fabb244bc902af3b9eab49ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:51.940786 301 raft_consensus.cc:1261] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35 [term 1 FOLLOWER]: Refusing update from remote peer e0390f7fabb244bc902af3b9eab49ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:51.941015 581 consensus_queue.cc:1035] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c39876dc73428d87aadbba9ed9de58" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:51.941152 574 consensus_queue.cc:1035] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "0513b0a4ecb94eb1935b1eda3d179c35" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:51.941900 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.941785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.2:43265)
0926 05:51:51.941827 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.941863 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 05:51:51.942135 577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.942001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.1:51075)
0926 05:51:51.942049 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.942074 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 05:51:51.943867 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.943676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.3:52009)
0926 05:51:51.943785 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.943824 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 05:51:51.944229 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.944146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.3:39255)
0926 05:51:51.944182 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.944205 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 05:51:51.944321 577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.944221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.1:50913)
0926 05:51:51.944254 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.944281 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 05:51:51.944685 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.944533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.24.255.2:43527)
0926 05:51:51.944630 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.944655 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 05:51:51.945448 586 mvcc.cc:205] Tried to move back new op lower bound from 7333486231306162176 to 7333486231021076480. Current Snapshot: MvccSnapshot[applied={T|T < 7333486231306162176}]
I20260926 05:51:51.946630 32498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:51.946710 32498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:51.948069 32498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebe661be221845cd986e688e7b4f6041: Sending DeleteTablet for 3 replicas of tablet 035a8860d2c446acbdfe320a70bcad0f
I20260926 05:51:51.948364 32749 tablet_service.cc:1514] Processing DeleteTablet for tablet 035a8860d2c446acbdfe320a70bcad0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:51 UTC) from {username='oryx', principal='oryx/127.24.255.60@KRBTEST.COM'} at 127.0.0.1:44910
I20260926 05:51:51.948417 32612 tablet_service.cc:1514] Processing DeleteTablet for tablet 035a8860d2c446acbdfe320a70bcad0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:51 UTC) from {username='oryx', principal='oryx/127.24.255.60@KRBTEST.COM'} at 127.0.0.1:48100
I20260926 05:51:51.948588 597 tablet_replica.cc:330] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35: stopping tablet replica
I20260926 05:51:51.948585 417 tablet_service.cc:1514] Processing DeleteTablet for tablet 035a8860d2c446acbdfe320a70bcad0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:51 UTC) from {username='oryx', principal='oryx/127.24.255.60@KRBTEST.COM'} at 127.0.0.1:39666
I20260926 05:51:51.948645 597 raft_consensus.cc:2229] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:51.948704 597 raft_consensus.cc:2258] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:51.948997 597 ts_tablet_manager.cc:1905] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:51.949002 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 32460
I20260926 05:51:51.951282 597 ts_tablet_manager.cc:1918] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:51.951352 597 log.cc:1196] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/035a8860d2c446acbdfe320a70bcad0f
I20260926 05:51:51.951596 597 ts_tablet_manager.cc:1939] T 035a8860d2c446acbdfe320a70bcad0f P 0513b0a4ecb94eb1935b1eda3d179c35: Deleting consensus metadata
I20260926 05:51:51.951747 599 tablet_replica.cc:330] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded: stopping tablet replica
I20260926 05:51:51.951813 599 raft_consensus.cc:2229] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:51:51.951918 599 raft_consensus.cc:2258] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:51.952036 32486 catalog_manager.cc:4841] TS 0513b0a4ecb94eb1935b1eda3d179c35 (127.24.255.2:37079): tablet 035a8860d2c446acbdfe320a70bcad0f (table test-table [id=b2b3da15175240d9adc03f42b2a16614]) successfully deleted
W20260926 05:51:51.954656 548 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:51.954511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:39449 (local address 127.0.0.1:48106)
0926 05:51:51.954558 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.954597 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:39449: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 05:51:51.954706 599 ts_tablet_manager.cc:1905] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 05:51:51.954803 32483 catalog_manager.cc:4584] TS e1c39876dc73428d87aadbba9ed9de58 (127.24.255.1:39449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 035a8860d2c446acbdfe320a70bcad0f: Network error: Client connection negotiation failed: client connection to 127.24.255.1:39449: connect: Connection reset by peer (error 104)
I20260926 05:51:51.954859 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 32686
I20260926 05:51:51.956934 599 ts_tablet_manager.cc:1918] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:51.957010 599 log.cc:1196] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/035a8860d2c446acbdfe320a70bcad0f
I20260926 05:51:51.957212 599 ts_tablet_manager.cc:1939] T 035a8860d2c446acbdfe320a70bcad0f P e0390f7fabb244bc902af3b9eab49ded: Deleting consensus metadata
I20260926 05:51:51.957624 32486 catalog_manager.cc:4841] TS e0390f7fabb244bc902af3b9eab49ded (127.24.255.3:46849): tablet 035a8860d2c446acbdfe320a70bcad0f (table test-table [id=b2b3da15175240d9adc03f42b2a16614]) successfully deleted
I20260926 05:51:51.957691 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.957507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:58068)
0926 05:51:51.957599 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.957630 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 05:51:51.957989 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:51.957881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46751 (local address 127.0.0.1:58070)
0926 05:51:51.957948 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:51.957972 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 05:51:51.961534 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 354
I20260926 05:51:51.967144 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 32180
I20260926 05:51:51.971441 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 32367
2026-09-26T05:51:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4920 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:52 dist-test-slave-7fr8 krb5kdc[602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:52 dist-test-slave-7fr8 krb5kdc[602](info): set up 2 sockets
Sep 26 05:51:52 dist-test-slave-7fr8 krb5kdc[602](info): commencing operation
krb5kdc: starting...
W20260926 05:51:52.950631 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:53 dist-test-slave-7fr8 krb5kdc[602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.62.keytab.
Entry for principal oryx/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.62.keytab.
I20260926 05:51:53.079563 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:34585
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.62.keytab
--principal=oryx/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:34585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:53.164162 615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:53.164392 615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:53.164417 615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:53.165624 615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:53.165669 615 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 05:51:53.165681 615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:53.165693 615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:53.165704 615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:53.165715 615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:53.167276 615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:34585
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.62.keytab
--principal=oryx/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:34585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:53.167500 615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:53.167686 615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:53.172904 623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:53.173089 615 server_base.cc:1034] running on GCE node
W20260926 05:51:53.172914 621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:53.172904 620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:53.173542 615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:53.173872 615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:53.175009 615 hybrid_clock.cc:648] HybridClock initialized: now 1790401913174997 us; error 30 us; skew 500 ppm
Sep 26 05:51:53 dist-test-slave-7fr8 krb5kdc[602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401913, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:53.177632 615 init.cc:377] Logged in from keytab as oryx/127.24.255.62@KRBTEST.COM (short username oryx)
I20260926 05:51:53.178088 615 webserver.cc:460] Webserver started at http://127.24.255.62:39115/ using document root <none> and password file <none>
I20260926 05:51:53.178318 615 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:53.178364 615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:53.178474 615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:53.179553 615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "5ee95393788043e58ef4623c85fa2993"
format_stamp: "Formatted at 2026-09-26 05:51:53 on dist-test-slave-7fr8"
server_key: "aa57bd094cf5a6aae79eb94b20db4583"
server_key_iv: "8c25c9581db1fae34d5323f2a09f27a1"
server_key_version: "encryptionkey@0"
I20260926 05:51:53.179894 615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "5ee95393788043e58ef4623c85fa2993"
format_stamp: "Formatted at 2026-09-26 05:51:53 on dist-test-slave-7fr8"
server_key: "aa57bd094cf5a6aae79eb94b20db4583"
server_key_iv: "8c25c9581db1fae34d5323f2a09f27a1"
server_key_version: "encryptionkey@0"
I20260926 05:51:53.181274 615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:53.182132 630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:53.182293 615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:53.182379 615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "5ee95393788043e58ef4623c85fa2993"
format_stamp: "Formatted at 2026-09-26 05:51:53 on dist-test-slave-7fr8"
server_key: "aa57bd094cf5a6aae79eb94b20db4583"
server_key_iv: "8c25c9581db1fae34d5323f2a09f27a1"
server_key_version: "encryptionkey@0"
I20260926 05:51:53.182439 615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:53.197372 615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:53.199643 615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:53.199765 615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:53.204329 615 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:34585
I20260926 05:51:53.204424 692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:34585 every 8 connection(s)
I20260926 05:51:53.204721 615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:53.205441 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:53.207890 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993: Bootstrap starting.
I20260926 05:51:53.208515 693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:53.208781 693 log.cc:824] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:53.209846 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993: No bootstrap required, opened a new log
I20260926 05:51:53.212888 693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee95393788043e58ef4623c85fa2993" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34585 } }
I20260926 05:51:53.213373 693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:53.213522 693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee95393788043e58ef4623c85fa2993, State: Initialized, Role: FOLLOWER
I20260926 05:51:53.213712 693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee95393788043e58ef4623c85fa2993" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34585 } }
I20260926 05:51:53.213809 693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:53.213843 693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:53.213894 693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:53.214668 693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee95393788043e58ef4623c85fa2993" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34585 } }
I20260926 05:51:53.214782 693 leader_election.cc:302] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ee95393788043e58ef4623c85fa2993; no voters:
I20260926 05:51:53.214895 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 615
I20260926 05:51:53.214988 693 leader_election.cc:288] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:53.215042 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:53.215034 696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:53.215150 696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee95393788043e58ef4623c85fa2993, State: Running, Role: LEADER
I20260926 05:51:53.215256 25596 external_mini_cluster.cc:1428] Setting key 807d972366df8c80cdb493610af16fa9
I20260926 05:51:53.215272 693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:53.215301 696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee95393788043e58ef4623c85fa2993" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34585 } }
I20260926 05:51:53.215866 698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ee95393788043e58ef4623c85fa2993" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee95393788043e58ef4623c85fa2993" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34585 } } }
I20260926 05:51:53.215870 697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ee95393788043e58ef4623c85fa2993. Latest consensus state: current_term: 1 leader_uuid: "5ee95393788043e58ef4623c85fa2993" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee95393788043e58ef4623c85fa2993" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34585 } } }
I20260926 05:51:53.216081 698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:53.216087 697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:53.216606 708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:53.217357 708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:53.218938 708 catalog_manager.cc:1347] Generated new cluster ID: 36b8a7850028463f93ba6d91603e88e7
I20260926 05:51:53.218983 708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:51:53 dist-test-slave-7fr8 krb5kdc[602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:53.228411 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.215777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57000 (local address 127.24.255.62:34585)
0926 05:51:53.216828 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.216840 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.216904 (+ 64us) server_negotiation.cc:407] Connection header received
0926 05:51:53.217040 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.217048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.217108 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.217257 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:53.218056 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.219480 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.220638 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.221364 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.221479 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.224368 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:53.224374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:53.224374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:53.224389 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:53.226635 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.227065 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.227068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.227069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.227193 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.227371 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.227372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.227372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.227725 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:53.227904 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.227977 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.228148 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000,"thread_start_us":62,"threads_started":1}
I20260926 05:51:53.237262 708 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:53.237802 708 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:53.241293 708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993: Generated new TSK 0
I20260926 05:51:53.241496 708 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab.
Entry for principal oryx/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab.
I20260926 05:51:53.273841 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34585
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab
--principal=oryx/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:53.360750 719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:53.360980 719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:53.361014 719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:53.362294 719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:53.362346 719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:53.362375 719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:51:53.362408 719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:53.364101 719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab
--principal=oryx/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.255.62:34585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:53.364380 719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:53.364614 719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:53.369513 725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:53.369517 724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:53.369517 727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:53.369798 719 server_base.cc:1034] running on GCE node
I20260926 05:51:53.370033 719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:53.370296 719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:53.371464 719 hybrid_clock.cc:648] HybridClock initialized: now 1790401913371450 us; error 29 us; skew 500 ppm
Sep 26 05:51:53 dist-test-slave-7fr8 krb5kdc[602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401913, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:53.374006 719 init.cc:377] Logged in from keytab as oryx/127.24.255.1@KRBTEST.COM (short username oryx)
I20260926 05:51:53.374459 719 webserver.cc:460] Webserver started at http://127.24.255.1:34099/ using document root <none> and password file <none>
I20260926 05:51:53.374676 719 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:53.374724 719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:53.374850 719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:53.375902 719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "0b8e33886c284db192cb2da2346880a1"
format_stamp: "Formatted at 2026-09-26 05:51:53 on dist-test-slave-7fr8"
server_key: "eff527d657806b0a66a330509fc828de"
server_key_iv: "4e7f0b369c01ae4511638c70e3dc5f3f"
server_key_version: "encryptionkey@0"
I20260926 05:51:53.376231 719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "0b8e33886c284db192cb2da2346880a1"
format_stamp: "Formatted at 2026-09-26 05:51:53 on dist-test-slave-7fr8"
server_key: "eff527d657806b0a66a330509fc828de"
server_key_iv: "4e7f0b369c01ae4511638c70e3dc5f3f"
server_key_version: "encryptionkey@0"
I20260926 05:51:53.377480 719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:51:53.378288 734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:53.378535 719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:53.378625 719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "0b8e33886c284db192cb2da2346880a1"
format_stamp: "Formatted at 2026-09-26 05:51:53 on dist-test-slave-7fr8"
server_key: "eff527d657806b0a66a330509fc828de"
server_key_iv: "4e7f0b369c01ae4511638c70e3dc5f3f"
server_key_version: "encryptionkey@0"
I20260926 05:51:53.378690 719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:53.405238 719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:53.407459 719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:53.407608 719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:53.408351 719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:53.408417 719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:53.408452 719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:53.408468 719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:51:53 dist-test-slave-7fr8 krb5kdc[602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401913, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:53.415563 719 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:45921
I20260926 05:51:53.415629 849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:45921 every 8 connection(s)
I20260926 05:51:53.415978 719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:51:53.418601 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.408941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:38169 (local address 127.24.255.62:34585)
0926 05:51:53.408994 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.408996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.409190 (+ 194us) server_negotiation.cc:407] Connection header received
0926 05:51:53.409686 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.409688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.409704 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.409772 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:53.410716 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.411292 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.412791 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.413054 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.413110 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.415821 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:53.415823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:53.415824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:53.415832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:53.417286 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.417662 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.417663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.417664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.417717 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.417933 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.417934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.417935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.418053 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:53.418129 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.418424 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.418511 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:51:53.418781 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.408920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34585 (local address 127.24.255.1:38169)
0926 05:51:53.409072 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:53.409107 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:53.409591 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:53.409877 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:53.409891 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:53.410289 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:53.410592 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.410600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.411368 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.411369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.412707 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.412711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.413138 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.413139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.413311 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.413889 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:53.413901 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:53.415704 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:53.417368 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:53.417375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:53.417377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:53.417594 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:53.417771 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:53.417772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:53.417772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:53.417878 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:53.418129 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:53.418135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:53.418282 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 05:51:53.418392 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 05:51:53.418566 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260926 05:51:53.419241 850 heartbeater.cc:347] Connected to a master server at 127.24.255.62:34585
I20260926 05:51:53.419324 850 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:53.419495 850 heartbeater.cc:511] Master 127.24.255.62:34585 requested a full tablet report, sending...
I20260926 05:51:53.419992 646 ts_manager.cc:194] Registered new tserver with Master: 0b8e33886c284db192cb2da2346880a1 (127.24.255.1:45921)
I20260926 05:51:53.420015 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 719
I20260926 05:51:53.420099 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:51:53.420317 25596 external_mini_cluster.cc:1428] Setting key c5df0dfc7daa41204c891a7ab5e202f4
I20260926 05:51:53.420868 646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:38169
I20260926 05:51:53.426007 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.420212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:60089 (local address 127.24.255.62:34585)
0926 05:51:53.420251 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.420253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.420344 (+ 91us) server_negotiation.cc:407] Connection header received
0926 05:51:53.420406 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.420407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.420416 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.420465 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:53.420768 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.421383 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.422069 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.422299 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.422355 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.423290 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:53.423291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:53.423362 (+ 71us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:53.423369 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:53.424728 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.425138 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.425139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.425139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.425199 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.425506 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.425508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.425508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.425628 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:53.425690 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.425822 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.425909 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:51:53.426030 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.420154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34585 (local address 127.24.255.1:60089)
0926 05:51:53.420284 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:53.420293 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:53.420347 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:53.420472 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:53.420473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:53.420645 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:53.420699 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.420700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.421463 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.421464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.421990 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.421992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.422359 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.422359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.422423 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.422894 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:53.422899 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:53.423199 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:53.424861 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:53.424862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:53.424862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:53.425049 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:53.425256 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:53.425257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:53.425257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:53.425380 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:53.425689 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:53.425689 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:53.425740 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 05:51:53.425794 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 05:51:53.425940 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.24.255.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab.
Entry for principal oryx/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab.
I20260926 05:51:53.445889 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34585
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab
--principal=oryx/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:53.529613 859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:53.529861 859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:53.529891 859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:53.531241 859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:53.531306 859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:53.531332 859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:51:53.531353 859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:53.532974 859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab
--principal=oryx/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.255.62:34585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:53.533215 859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:53.533443 859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:53.538853 867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:53.538880 864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:53.538853 865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:53.538877 859 server_base.cc:1034] running on GCE node
I20260926 05:51:53.539314 859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:53.539566 859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:53.540725 859 hybrid_clock.cc:648] HybridClock initialized: now 1790401913540707 us; error 35 us; skew 500 ppm
Sep 26 05:51:53 dist-test-slave-7fr8 krb5kdc[602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401913, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:53.543282 859 init.cc:377] Logged in from keytab as oryx/127.24.255.2@KRBTEST.COM (short username oryx)
I20260926 05:51:53.543761 859 webserver.cc:460] Webserver started at http://127.24.255.2:38319/ using document root <none> and password file <none>
I20260926 05:51:53.543953 859 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:53.543993 859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:53.544085 859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:53.545097 859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "0530fe425ad54b7f83e44f18571ea8e2"
format_stamp: "Formatted at 2026-09-26 05:51:53 on dist-test-slave-7fr8"
server_key: "f2be9c3e4a2bc2fb9cfc7548361891fa"
server_key_iv: "d9e2ca69206b8908345ed3e7448766ef"
server_key_version: "encryptionkey@0"
I20260926 05:51:53.545390 859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "0530fe425ad54b7f83e44f18571ea8e2"
format_stamp: "Formatted at 2026-09-26 05:51:53 on dist-test-slave-7fr8"
server_key: "f2be9c3e4a2bc2fb9cfc7548361891fa"
server_key_iv: "d9e2ca69206b8908345ed3e7448766ef"
server_key_version: "encryptionkey@0"
I20260926 05:51:53.546798 859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 05:51:53.547744 874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:53.548084 859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:53.548190 859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "0530fe425ad54b7f83e44f18571ea8e2"
format_stamp: "Formatted at 2026-09-26 05:51:53 on dist-test-slave-7fr8"
server_key: "f2be9c3e4a2bc2fb9cfc7548361891fa"
server_key_iv: "d9e2ca69206b8908345ed3e7448766ef"
server_key_version: "encryptionkey@0"
I20260926 05:51:53.548272 859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:53.565805 859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:53.568321 859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:53.568527 859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:53.569229 859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:53.569283 859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:53.569435 859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:53.569492 859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:51:53 dist-test-slave-7fr8 krb5kdc[602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401913, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:53.576493 859 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:46623
I20260926 05:51:53.576535 989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:46623 every 8 connection(s)
I20260926 05:51:53.577271 859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:51:53.578912 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.569531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:38309 (local address 127.24.255.62:34585)
0926 05:51:53.569593 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.569595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.569839 (+ 244us) server_negotiation.cc:407] Connection header received
0926 05:51:53.570385 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.570386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.570399 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.570456 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:53.571102 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.571593 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.572625 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.572813 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.572896 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.576065 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:53.576067 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:53.576067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:53.576075 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:53.577524 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.577965 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.577966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.577967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.578028 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.578271 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.578272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.578273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.578400 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:53.578457 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.578727 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.578812 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:51:53.579062 883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.569531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34585 (local address 127.24.255.2:38309)
0926 05:51:53.569710 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:53.569750 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:53.570286 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:53.570525 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:53.570529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:53.570768 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:53.571018 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.571023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.571677 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.571677 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.572542 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.572545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.572908 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.572909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.573027 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.573570 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:53.573581 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:53.575947 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:53.577615 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:53.577621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:53.577624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:53.577891 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:53.578080 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:53.578081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:53.578082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:53.578215 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:53.578453 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:53.578459 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:53.578599 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 05:51:53.578680 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:51:53.578844 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
I20260926 05:51:53.579531 990 heartbeater.cc:347] Connected to a master server at 127.24.255.62:34585
I20260926 05:51:53.579617 990 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:53.579794 990 heartbeater.cc:511] Master 127.24.255.62:34585 requested a full tablet report, sending...
I20260926 05:51:53.580183 646 ts_manager.cc:194] Registered new tserver with Master: 0530fe425ad54b7f83e44f18571ea8e2 (127.24.255.2:46623)
I20260926 05:51:53.580659 646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:38309
I20260926 05:51:53.581106 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 859
I20260926 05:51:53.581199 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:51:53.581399 25596 external_mini_cluster.cc:1428] Setting key d894b6146001e8d1b6d65f621c32bbd0
I20260926 05:51:53.586737 996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.580503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34585 (local address 127.24.255.2:43879)
0926 05:51:53.580657 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:53.580666 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:53.580718 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:53.580829 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:53.580830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:53.581016 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:53.581098 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.581100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.582029 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.582030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.582669 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.582671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.583006 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.583007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.583077 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.583594 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:53.583600 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:53.583986 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:53.585637 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:53.585638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:53.585638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:53.585850 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:53.586057 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:53.586058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:53.586058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:53.586144 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:53.586409 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:53.586410 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:53.586459 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 05:51:53.586517 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 05:51:53.586627 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260926 05:51:53.586725 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.580551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:43879 (local address 127.24.255.62:34585)
0926 05:51:53.580590 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.580592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.580715 (+ 123us) server_negotiation.cc:407] Connection header received
0926 05:51:53.580772 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.580773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.580782 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.580826 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:53.581207 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.581846 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.582738 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.582945 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.582999 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.584105 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:53.584107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:53.584107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:53.584114 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:53.585544 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.585939 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.585940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.585941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.586004 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.586188 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.586189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.586189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.586340 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:53.586402 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.586545 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.586626 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for oryx/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab.
Entry for principal oryx/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab.
I20260926 05:51:53.607241 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34585
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab
--principal=oryx/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:53.692939 999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:53.693171 999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:53.693198 999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:53.694463 999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:53.694517 999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:53.694538 999 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:51:53.694555 999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:53.696156 999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab
--principal=oryx/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.255.62:34585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:53.696391 999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:53.696602 999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:53.701930 1004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:53.701938 1007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:53.702128 1005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:53.702235 999 server_base.cc:1034] running on GCE node
I20260926 05:51:53.702456 999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:53.702771 999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:53.703938 999 hybrid_clock.cc:648] HybridClock initialized: now 1790401913703923 us; error 27 us; skew 500 ppm
Sep 26 05:51:53 dist-test-slave-7fr8 krb5kdc[602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401913, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:53.706522 999 init.cc:377] Logged in from keytab as oryx/127.24.255.3@KRBTEST.COM (short username oryx)
I20260926 05:51:53.706970 999 webserver.cc:460] Webserver started at http://127.24.255.3:43653/ using document root <none> and password file <none>
I20260926 05:51:53.707207 999 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:53.707262 999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:53.707378 999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:53.708470 999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "76f17271a0344c7d8b7d985759c31b18"
format_stamp: "Formatted at 2026-09-26 05:51:53 on dist-test-slave-7fr8"
server_key: "8df32c884ce96a0fa2586a70f625fe19"
server_key_iv: "41807c7c57c8be2ee3caaff5f7856ef8"
server_key_version: "encryptionkey@0"
I20260926 05:51:53.708783 999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "76f17271a0344c7d8b7d985759c31b18"
format_stamp: "Formatted at 2026-09-26 05:51:53 on dist-test-slave-7fr8"
server_key: "8df32c884ce96a0fa2586a70f625fe19"
server_key_iv: "41807c7c57c8be2ee3caaff5f7856ef8"
server_key_version: "encryptionkey@0"
I20260926 05:51:53.710143 999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:51:53.710932 1014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:53.711143 999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:53.711234 999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "76f17271a0344c7d8b7d985759c31b18"
format_stamp: "Formatted at 2026-09-26 05:51:53 on dist-test-slave-7fr8"
server_key: "8df32c884ce96a0fa2586a70f625fe19"
server_key_iv: "41807c7c57c8be2ee3caaff5f7856ef8"
server_key_version: "encryptionkey@0"
I20260926 05:51:53.711310 999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:53.724298 999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:53.726665 999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:53.726805 999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:53.727625 999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:53.727674 999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:53.727710 999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:53.727739 999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:51:53 dist-test-slave-7fr8 krb5kdc[602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401913, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:53.735314 999 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:33373
I20260926 05:51:53.735460 1129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:33373 every 8 connection(s)
I20260926 05:51:53.735747 999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:51:53.737902 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.728008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:58233 (local address 127.24.255.62:34585)
0926 05:51:53.728074 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.728077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.728268 (+ 191us) server_negotiation.cc:407] Connection header received
0926 05:51:53.728877 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.728878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.728893 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.728981 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:53.729905 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.730719 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.731788 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.732075 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.732157 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.734836 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:53.734841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:53.734842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:53.734855 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:53.736560 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.736991 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.736993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.736993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.737061 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.737273 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.737273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.737274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.737388 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:53.737440 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.737762 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.737837 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:51:53.738021 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.727987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34585 (local address 127.24.255.3:58233)
0926 05:51:53.728141 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:53.728183 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:53.728775 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:53.729121 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:53.729125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:53.729494 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:53.729771 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.729777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.730836 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.730837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.731701 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.731704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.732195 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.732196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.732340 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.733006 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:53.733014 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:53.734667 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:53.736661 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:53.736666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:53.736668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:53.736895 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:53.737123 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:53.737124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:53.737124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:53.737219 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:53.737471 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:53.737477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:53.737607 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 05:51:53.737705 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 05:51:53.737845 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260926 05:51:53.738492 1130 heartbeater.cc:347] Connected to a master server at 127.24.255.62:34585
I20260926 05:51:53.738572 1130 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:53.738742 1130 heartbeater.cc:511] Master 127.24.255.62:34585 requested a full tablet report, sending...
I20260926 05:51:53.739120 647 ts_manager.cc:194] Registered new tserver with Master: 76f17271a0344c7d8b7d985759c31b18 (127.24.255.3:33373)
I20260926 05:51:53.739492 647 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:58233
I20260926 05:51:53.743072 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 999
I20260926 05:51:53.743176 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:51:53.743399 25596 external_mini_cluster.cc:1428] Setting key a7d906a266c340258872405adc0fd433
I20260926 05:51:53.744475 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:51:53.745360 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.739534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:35961 (local address 127.24.255.62:34585)
0926 05:51:53.739580 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.739581 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.739696 (+ 115us) server_negotiation.cc:407] Connection header received
0926 05:51:53.739750 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.739751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.739761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.739804 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:53.740104 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.740577 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.741363 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.741561 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.741615 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.742730 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:53.742731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:53.742731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:53.742739 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:53.744251 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.744677 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.744678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.744678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.744731 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.744925 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.744925 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.744926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.745029 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:53.745074 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.745233 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.745295 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:51:53.745450 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.739457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34585 (local address 127.24.255.3:35961)
0926 05:51:53.739631 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:53.739640 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:53.739696 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:53.739807 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:53.739807 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:53.739977 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:53.740041 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.740043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.740659 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.740660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.741273 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.741276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.741622 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.741622 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.741712 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.742321 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:53.742327 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:53.742627 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:53.744363 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:53.744364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:53.744365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:53.744607 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:53.744792 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:53.744793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:53.744793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:53.744869 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:53.745082 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:53.745083 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:53.745137 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 05:51:53.745219 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 05:51:53.745348 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
Sep 26 05:51:53 dist-test-slave-7fr8 krb5kdc[602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401913, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:51:53 dist-test-slave-7fr8 krb5kdc[602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:53 dist-test-slave-7fr8 krb5kdc[602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 05:51:53.766611 704 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:53.760690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.24.255.62:34585)
0926 05:51:53.760748 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.760750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.760759 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:53.760805 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.760806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.760815 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.760862 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:53.761576 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.762087 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.762974 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.763215 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.763279 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.766530 (+ 3251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57010: BlockingRecv error: recv got EOF from 127.0.0.1:57010 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 05:51:53 dist-test-slave-7fr8 krb5kdc[602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401913, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:53.773705 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.767401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.24.255.62:34585)
0926 05:51:53.767468 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.767469 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.767478 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:53.767501 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.767501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.767510 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.767563 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:53.768002 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.768507 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.769196 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.769389 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.769446 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.771645 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:53.771647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:53.771647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:53.771655 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:53.772751 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.773052 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.773053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.773054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.773102 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:53.773251 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:53.773252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:53.773252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:53.773358 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:53.773446 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.773537 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.773613 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:51:53.775409 647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:51:53.776994 647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:53.783699 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.779416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56588 (local address 127.24.255.2:46623)
0926 05:51:53.779670 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.779675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.779686 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:51:53.779726 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.779728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.779745 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.779827 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:53.780034 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.781119 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.782666 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.783183 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.783220 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.783274 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.783541 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.783589 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":105,"threads_started":1}
I20260926 05:51:53.783687 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.779289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:46623 (local address 127.0.0.1:56588)
0926 05:51:53.779529 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:53.779569 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:53.779637 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:53.779849 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:53.779852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:53.779854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:53.779940 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.779943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.781211 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.781212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.782565 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.782568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.783331 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.783332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.783430 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.783440 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 05:51:53.783512 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:51:53.783584 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":38,"thread_start_us":70,"threads_started":1}
I20260926 05:51:53.783982 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.779428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.24.255.3:33373)
0926 05:51:53.779642 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.779648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.779901 (+ 253us) server_negotiation.cc:407] Connection header received
0926 05:51:53.779942 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.779944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.779959 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.780025 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:53.780202 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.781278 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.782669 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.783350 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.783409 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.783471 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.783726 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.783884 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260926 05:51:53.783984 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.779790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.24.255.1:45921)
0926 05:51:53.779977 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.779981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.779992 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:51:53.780054 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.780056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.780072 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.780147 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:53.780261 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.781371 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.782536 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.783247 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.783321 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.783404 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.783852 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.783893 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260926 05:51:53.783689 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.779580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:45921 (local address 127.0.0.1:47300)
0926 05:51:53.779901 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:53.779951 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:53.779992 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:53.780152 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:53.780153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:53.780154 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:53.780184 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.780185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.781433 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.781434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.782454 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.782456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.783337 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.783338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.783426 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.783439 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 05:51:53.783511 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:51:53.783584 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":31,"thread_start_us":76,"threads_started":1}
I20260926 05:51:53.784257 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.779320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:33373 (local address 127.0.0.1:58080)
0926 05:51:53.779792 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:53.779801 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:53.779847 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:53.780078 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:53.780079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:53.780081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:53.780131 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.780133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.781366 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.781367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.782608 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.782609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.783504 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.783505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.783567 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.783570 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:51:53.783616 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:51:53.783657 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":2,"thread_start_us":146,"threads_started":1}
I20260926 05:51:53.784458 924 tablet_service.cc:1467] Processing CreateTablet for tablet c776e2f6cc4c4953b3f645af74389c16 (DEFAULT_TABLE table=test-table [id=75ec9be7b7db44039f6b88655059b826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:53.784950 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c776e2f6cc4c4953b3f645af74389c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:53.785099 784 tablet_service.cc:1467] Processing CreateTablet for tablet c776e2f6cc4c4953b3f645af74389c16 (DEFAULT_TABLE table=test-table [id=75ec9be7b7db44039f6b88655059b826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:53.785384 784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c776e2f6cc4c4953b3f645af74389c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:53.786799 1064 tablet_service.cc:1467] Processing CreateTablet for tablet c776e2f6cc4c4953b3f645af74389c16 (DEFAULT_TABLE table=test-table [id=75ec9be7b7db44039f6b88655059b826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:53.787151 1064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c776e2f6cc4c4953b3f645af74389c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:53.788921 1158 tablet_bootstrap.cc:492] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1: Bootstrap starting.
I20260926 05:51:53.789348 1159 tablet_bootstrap.cc:492] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2: Bootstrap starting.
I20260926 05:51:53.789666 1158 tablet_bootstrap.cc:654] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:53.789927 1159 tablet_bootstrap.cc:654] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:53.789990 1158 log.cc:824] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:53.790258 1159 log.cc:824] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:53.790784 1158 tablet_bootstrap.cc:492] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1: No bootstrap required, opened a new log
I20260926 05:51:53.790859 1158 ts_tablet_manager.cc:1397] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:51:53.790877 1159 tablet_bootstrap.cc:492] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2: No bootstrap required, opened a new log
I20260926 05:51:53.790925 1159 ts_tablet_manager.cc:1397] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:51:53.792824 1159 raft_consensus.cc:348] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } }
I20260926 05:51:53.792816 1158 raft_consensus.cc:348] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } }
I20260926 05:51:53.793004 1158 raft_consensus.cc:374] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:53.793004 1159 raft_consensus.cc:374] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:53.793040 1159 raft_consensus.cc:729] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0530fe425ad54b7f83e44f18571ea8e2, State: Initialized, Role: FOLLOWER
I20260926 05:51:53.793040 1158 raft_consensus.cc:729] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8e33886c284db192cb2da2346880a1, State: Initialized, Role: FOLLOWER
I20260926 05:51:53.793155 1159 consensus_queue.cc:260] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } }
I20260926 05:51:53.793156 1158 consensus_queue.cc:260] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } }
I20260926 05:51:53.793442 1158 ts_tablet_manager.cc:1428] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:51:53.793442 1159 ts_tablet_manager.cc:1428] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 05:51:53.793481 990 heartbeater.cc:503] Master 127.24.255.62:34585 was elected leader, sending a full tablet report...
I20260926 05:51:53.793576 850 heartbeater.cc:503] Master 127.24.255.62:34585 was elected leader, sending a full tablet report...
I20260926 05:51:53.793690 1161 tablet_bootstrap.cc:492] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18: Bootstrap starting.
I20260926 05:51:53.794413 1161 tablet_bootstrap.cc:654] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:53.794761 1161 log.cc:824] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:53.795799 1161 tablet_bootstrap.cc:492] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18: No bootstrap required, opened a new log
I20260926 05:51:53.795883 1161 ts_tablet_manager.cc:1397] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:51:53.797325 1161 raft_consensus.cc:348] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } }
I20260926 05:51:53.797473 1161 raft_consensus.cc:374] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:53.797514 1161 raft_consensus.cc:729] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f17271a0344c7d8b7d985759c31b18, State: Initialized, Role: FOLLOWER
I20260926 05:51:53.797638 1161 consensus_queue.cc:260] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } }
I20260926 05:51:53.797925 1161 ts_tablet_manager.cc:1428] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:51:53.798063 1130 heartbeater.cc:503] Master 127.24.255.62:34585 was elected leader, sending a full tablet report...
W20260926 05:51:53.828018 991 tablet.cc:2369] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:51:53.917092 851 tablet.cc:2369] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:51:53.919488 1166 raft_consensus.cc:482] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:53.919730 1166 raft_consensus.cc:504] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } }
I20260926 05:51:53.920219 1166 leader_election.cc:288] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0530fe425ad54b7f83e44f18571ea8e2 (127.24.255.2:46623), 0b8e33886c284db192cb2da2346880a1 (127.24.255.1:45921)
I20260926 05:51:53.924283 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.920572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:38105 (local address 127.24.255.1:45921)
0926 05:51:53.920619 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.920621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.920769 (+ 148us) server_negotiation.cc:407] Connection header received
0926 05:51:53.920826 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.920827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.920837 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.920978 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:53.921157 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.921845 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.923298 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.923867 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.923947 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.923956 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.924164 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.924212 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:51:53.924295 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.920467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:37473 (local address 127.24.255.2:46623)
0926 05:51:53.920555 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.920556 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.920566 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:53.920594 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.920595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.920604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.920659 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:53.920859 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.921774 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.923312 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.923885 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.923947 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.923956 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.924196 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.924238 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 05:51:53.924327 1167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.920419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:45921 (local address 127.24.255.3:38105)
0926 05:51:53.920665 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:53.920680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:53.920765 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:53.920987 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:53.920988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:53.920990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:53.921053 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.921054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.921926 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.921926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.923194 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.923195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.923973 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.923974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.924081 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.924086 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:51:53.924149 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 05:51:53.924199 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":50,"thread_start_us":110,"threads_started":1}
I20260926 05:51:53.924333 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.920416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:46623 (local address 127.24.255.3:37473)
0926 05:51:53.920471 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:53.920484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:53.920528 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:53.920710 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:53.920712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:53.920714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:53.920789 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.920790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.921897 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.921898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.923182 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:53.923186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.923989 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.923990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:53.924084 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.924086 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:51:53.924149 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 05:51:53.924199 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":5}
I20260926 05:51:53.924708 804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c776e2f6cc4c4953b3f645af74389c16" candidate_uuid: "76f17271a0344c7d8b7d985759c31b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8e33886c284db192cb2da2346880a1" is_pre_election: true
I20260926 05:51:53.924850 804 raft_consensus.cc:2454] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76f17271a0344c7d8b7d985759c31b18 in term 0.
I20260926 05:51:53.924943 944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c776e2f6cc4c4953b3f645af74389c16" candidate_uuid: "76f17271a0344c7d8b7d985759c31b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0530fe425ad54b7f83e44f18571ea8e2" is_pre_election: true
I20260926 05:51:53.925088 944 raft_consensus.cc:2454] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76f17271a0344c7d8b7d985759c31b18 in term 0.
I20260926 05:51:53.925068 1016 leader_election.cc:302] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8e33886c284db192cb2da2346880a1, 76f17271a0344c7d8b7d985759c31b18; no voters:
I20260926 05:51:53.925205 1166 raft_consensus.cc:2781] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:53.925278 1166 raft_consensus.cc:482] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:53.925320 1166 raft_consensus.cc:3037] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:53.926277 1166 raft_consensus.cc:504] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } }
I20260926 05:51:53.926436 1166 leader_election.cc:288] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [CANDIDATE]: Term 1 election: Requested vote from peers 0530fe425ad54b7f83e44f18571ea8e2 (127.24.255.2:46623), 0b8e33886c284db192cb2da2346880a1 (127.24.255.1:45921)
I20260926 05:51:53.926589 944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c776e2f6cc4c4953b3f645af74389c16" candidate_uuid: "76f17271a0344c7d8b7d985759c31b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0530fe425ad54b7f83e44f18571ea8e2"
I20260926 05:51:53.926604 804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c776e2f6cc4c4953b3f645af74389c16" candidate_uuid: "76f17271a0344c7d8b7d985759c31b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8e33886c284db192cb2da2346880a1"
I20260926 05:51:53.926664 944 raft_consensus.cc:3037] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:53.926678 804 raft_consensus.cc:3037] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:53.927348 944 raft_consensus.cc:2454] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76f17271a0344c7d8b7d985759c31b18 in term 1.
I20260926 05:51:53.927407 804 raft_consensus.cc:2454] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76f17271a0344c7d8b7d985759c31b18 in term 1.
I20260926 05:51:53.927553 1017 leader_election.cc:302] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0530fe425ad54b7f83e44f18571ea8e2, 76f17271a0344c7d8b7d985759c31b18; no voters:
I20260926 05:51:53.927659 1166 raft_consensus.cc:2781] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:53.927816 1166 raft_consensus.cc:686] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [term 1 LEADER]: Becoming Leader. State: Replica: 76f17271a0344c7d8b7d985759c31b18, State: Running, Role: LEADER
I20260926 05:51:53.927910 1166 consensus_queue.cc:237] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } }
I20260926 05:51:53.928893 647 catalog_manager.cc:5495] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76f17271a0344c7d8b7d985759c31b18 (127.24.255.3). New cstate: current_term: 1 leader_uuid: "76f17271a0344c7d8b7d985759c31b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } health_report { overall_health: UNKNOWN } } }
I20260926 05:51:53.935752 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:53.933447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.24.255.3:33373)
0926 05:51:53.933533 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:53.933535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:53.933544 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:53.933572 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:53.933573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:53.933589 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:53.933646 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:53.933771 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.934306 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.935051 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:53.935251 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:53.935304 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:53.935444 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:53.935551 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:53.935619 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:53.935654 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 05:51:53.935675 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 05:51:53.937594 944 raft_consensus.cc:1261] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Refusing update from remote peer 76f17271a0344c7d8b7d985759c31b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:53.937655 804 raft_consensus.cc:1261] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Refusing update from remote peer 76f17271a0344c7d8b7d985759c31b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:53.937959 1168 consensus_queue.cc:1035] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:53.938130 1166 consensus_queue.cc:1035] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:53.940618 1171 mvcc.cc:205] Tried to move back new op lower bound from 7333486239485337600 to 7333486239449448448. Current Snapshot: MvccSnapshot[applied={T|T < 7333486239485337600}]
I20260926 05:51:53.943966 646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:53.944103 646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:53.945070 646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993: Sending DeleteTablet for 3 replicas of tablet c776e2f6cc4c4953b3f645af74389c16
I20260926 05:51:53.945446 924 tablet_service.cc:1514] Processing DeleteTablet for tablet c776e2f6cc4c4953b3f645af74389c16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:53 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:56588
I20260926 05:51:53.945474 1064 tablet_service.cc:1514] Processing DeleteTablet for tablet c776e2f6cc4c4953b3f645af74389c16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:53 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:58080
I20260926 05:51:53.945463 784 tablet_service.cc:1514] Processing DeleteTablet for tablet c776e2f6cc4c4953b3f645af74389c16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:53 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:47300
I20260926 05:51:53.945923 1181 tablet_replica.cc:330] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1: stopping tablet replica
I20260926 05:51:53.945941 1183 tablet_replica.cc:330] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2: stopping tablet replica
I20260926 05:51:53.946056 1183 raft_consensus.cc:2229] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:53.946079 1182 tablet_replica.cc:330] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18: stopping tablet replica
I20260926 05:51:53.946030 646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:51:53.946204 646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:53.946313 1181 raft_consensus.cc:2229] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:53.946331 1183 raft_consensus.cc:2258] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:53.946422 1181 raft_consensus.cc:2258] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:53.946797 1183 ts_tablet_manager.cc:1905] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:53.946574 1182 raft_consensus.cc:2229] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:51:53.946976 1182 raft_consensus.cc:2258] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:53.947521 1182 ts_tablet_manager.cc:1905] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:53.947728 784 tablet_service.cc:1467] Processing CreateTablet for tablet cfad9485fba94f56be24a4619ab371ee (DEFAULT_TABLE table=test-table [id=77f169915134491dba51acb841975e35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:53.947836 784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfad9485fba94f56be24a4619ab371ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:53.948506 1183 ts_tablet_manager.cc:1918] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:53.948573 1183 log.cc:1196] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/c776e2f6cc4c4953b3f645af74389c16
I20260926 05:51:53.948799 1183 ts_tablet_manager.cc:1939] T c776e2f6cc4c4953b3f645af74389c16 P 0530fe425ad54b7f83e44f18571ea8e2: Deleting consensus metadata
I20260926 05:51:53.949122 1158 tablet_bootstrap.cc:492] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1: Bootstrap starting.
I20260926 05:51:53.949142 1182 ts_tablet_manager.cc:1918] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:53.949203 1182 log.cc:1196] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/c776e2f6cc4c4953b3f645af74389c16
I20260926 05:51:53.949417 924 tablet_service.cc:1467] Processing CreateTablet for tablet cfad9485fba94f56be24a4619ab371ee (DEFAULT_TABLE table=test-table [id=77f169915134491dba51acb841975e35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:53.949436 1182 ts_tablet_manager.cc:1939] T c776e2f6cc4c4953b3f645af74389c16 P 76f17271a0344c7d8b7d985759c31b18: Deleting consensus metadata
I20260926 05:51:53.949474 1158 tablet_bootstrap.cc:654] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:53.949514 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfad9485fba94f56be24a4619ab371ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:53.950047 633 catalog_manager.cc:4841] TS 0530fe425ad54b7f83e44f18571ea8e2 (127.24.255.2:46623): tablet c776e2f6cc4c4953b3f645af74389c16 (table test-table [id=75ec9be7b7db44039f6b88655059b826]) successfully deleted
I20260926 05:51:53.950119 1158 tablet_bootstrap.cc:492] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1: No bootstrap required, opened a new log
I20260926 05:51:53.950166 1158 ts_tablet_manager.cc:1397] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:53.950302 1158 raft_consensus.cc:348] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:53.950371 1158 raft_consensus.cc:374] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:53.950389 1158 raft_consensus.cc:729] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8e33886c284db192cb2da2346880a1, State: Initialized, Role: FOLLOWER
I20260926 05:51:53.950433 1158 consensus_queue.cc:260] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:53.950544 1158 ts_tablet_manager.cc:1428] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:53.950598 1159 tablet_bootstrap.cc:492] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2: Bootstrap starting.
I20260926 05:51:53.950971 1159 tablet_bootstrap.cc:654] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:53.951156 632 catalog_manager.cc:4841] TS 76f17271a0344c7d8b7d985759c31b18 (127.24.255.3:33373): tablet c776e2f6cc4c4953b3f645af74389c16 (table test-table [id=75ec9be7b7db44039f6b88655059b826]) successfully deleted
I20260926 05:51:53.951268 1064 tablet_service.cc:1467] Processing CreateTablet for tablet cfad9485fba94f56be24a4619ab371ee (DEFAULT_TABLE table=test-table [id=77f169915134491dba51acb841975e35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:53.951370 1064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfad9485fba94f56be24a4619ab371ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:53.951594 1159 tablet_bootstrap.cc:492] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2: No bootstrap required, opened a new log
I20260926 05:51:53.951629 1159 ts_tablet_manager.cc:1397] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:53.951759 1159 raft_consensus.cc:348] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:53.951819 1159 raft_consensus.cc:374] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:53.951836 1159 raft_consensus.cc:729] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0530fe425ad54b7f83e44f18571ea8e2, State: Initialized, Role: FOLLOWER
I20260926 05:51:53.951877 1159 consensus_queue.cc:260] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:53.951960 1159 ts_tablet_manager.cc:1428] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:53.952553 1161 tablet_bootstrap.cc:492] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18: Bootstrap starting.
I20260926 05:51:53.952953 1161 tablet_bootstrap.cc:654] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:53.953240 1181 ts_tablet_manager.cc:1905] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:53.953557 1161 tablet_bootstrap.cc:492] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18: No bootstrap required, opened a new log
I20260926 05:51:53.953608 1161 ts_tablet_manager.cc:1397] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:53.953745 1161 raft_consensus.cc:348] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:53.953811 1161 raft_consensus.cc:374] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:53.953830 1161 raft_consensus.cc:729] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f17271a0344c7d8b7d985759c31b18, State: Initialized, Role: FOLLOWER
I20260926 05:51:53.953881 1161 consensus_queue.cc:260] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:53.953964 1161 ts_tablet_manager.cc:1428] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:53.955029 1181 ts_tablet_manager.cc:1918] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:53.955107 1181 log.cc:1196] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/wal/wals/c776e2f6cc4c4953b3f645af74389c16
I20260926 05:51:53.955631 1181 ts_tablet_manager.cc:1939] T c776e2f6cc4c4953b3f645af74389c16 P 0b8e33886c284db192cb2da2346880a1: Deleting consensus metadata
I20260926 05:51:53.956004 632 catalog_manager.cc:4841] TS 0b8e33886c284db192cb2da2346880a1 (127.24.255.1:45921): tablet c776e2f6cc4c4953b3f645af74389c16 (table test-table [id=75ec9be7b7db44039f6b88655059b826]) successfully deleted
W20260926 05:51:53.986320 1131 tablet.cc:2369] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:51:54.174531 1163 raft_consensus.cc:482] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:54.174645 1163 raft_consensus.cc:504] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.174960 1163 leader_election.cc:288] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b8e33886c284db192cb2da2346880a1 (127.24.255.1:45921), 76f17271a0344c7d8b7d985759c31b18 (127.24.255.3:33373)
I20260926 05:51:54.177863 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.175378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:35585 (local address 127.24.255.1:45921)
0926 05:51:54.175423 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:54.175425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:54.175436 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:51:54.175563 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:54.175564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:54.175573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:54.175622 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:54.175762 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.176256 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.177207 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.177558 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.177619 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.177628 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:54.177754 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 05:51:54.177789 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:51:54.177891 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.175419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:44035 (local address 127.24.255.3:33373)
0926 05:51:54.175473 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:54.175474 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:54.175526 (+ 52us) server_negotiation.cc:407] Connection header received
0926 05:51:54.175589 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:54.175590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:54.175601 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:54.175647 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:54.175769 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.176232 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.177235 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.177602 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.177663 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.177671 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:54.177791 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 05:51:54.177825 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:51:54.177908 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.175192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:45921 (local address 127.24.255.2:35585)
0926 05:51:54.175363 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:54.175376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:54.175467 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:54.175630 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:54.175631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:54.175633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:54.175706 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.175707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.176335 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.176336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.177122 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.177124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.177625 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.177626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.177695 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.177699 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:51:54.177745 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:51:54.177788 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":57,"threads_started":1}
I20260926 05:51:54.177919 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.175326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:33373 (local address 127.24.255.2:44035)
0926 05:51:54.175438 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:54.175448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:54.175529 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:54.175655 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:54.175656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:54.175658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:54.175716 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.175717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.176303 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.176304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.177150 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.177152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.177668 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.177669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.177733 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.177736 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:51:54.177779 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 05:51:54.177823 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":44,"thread_start_us":50,"threads_started":1}
I20260926 05:51:54.178260 804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfad9485fba94f56be24a4619ab371ee" candidate_uuid: "0530fe425ad54b7f83e44f18571ea8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8e33886c284db192cb2da2346880a1" is_pre_election: true
I20260926 05:51:54.178248 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfad9485fba94f56be24a4619ab371ee" candidate_uuid: "0530fe425ad54b7f83e44f18571ea8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f17271a0344c7d8b7d985759c31b18" is_pre_election: true
I20260926 05:51:54.178375 804 raft_consensus.cc:2454] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0530fe425ad54b7f83e44f18571ea8e2 in term 0.
I20260926 05:51:54.178391 1084 raft_consensus.cc:2454] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0530fe425ad54b7f83e44f18571ea8e2 in term 0.
I20260926 05:51:54.178542 876 leader_election.cc:302] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0530fe425ad54b7f83e44f18571ea8e2, 0b8e33886c284db192cb2da2346880a1; no voters:
I20260926 05:51:54.178683 1163 raft_consensus.cc:2781] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:54.178732 1163 raft_consensus.cc:482] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:54.178746 1163 raft_consensus.cc:3037] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:54.179466 1163 raft_consensus.cc:504] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.179590 1163 leader_election.cc:288] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [CANDIDATE]: Term 1 election: Requested vote from peers 0b8e33886c284db192cb2da2346880a1 (127.24.255.1:45921), 76f17271a0344c7d8b7d985759c31b18 (127.24.255.3:33373)
I20260926 05:51:54.179761 804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfad9485fba94f56be24a4619ab371ee" candidate_uuid: "0530fe425ad54b7f83e44f18571ea8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8e33886c284db192cb2da2346880a1"
I20260926 05:51:54.179790 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfad9485fba94f56be24a4619ab371ee" candidate_uuid: "0530fe425ad54b7f83e44f18571ea8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f17271a0344c7d8b7d985759c31b18"
I20260926 05:51:54.179836 804 raft_consensus.cc:3037] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:54.179862 1084 raft_consensus.cc:3037] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:54.180424 804 raft_consensus.cc:2454] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0530fe425ad54b7f83e44f18571ea8e2 in term 1.
I20260926 05:51:54.180598 876 leader_election.cc:302] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0530fe425ad54b7f83e44f18571ea8e2, 0b8e33886c284db192cb2da2346880a1; no voters:
I20260926 05:51:54.180609 1084 raft_consensus.cc:2454] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0530fe425ad54b7f83e44f18571ea8e2 in term 1.
I20260926 05:51:54.180675 1163 raft_consensus.cc:2781] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:54.180809 1163 raft_consensus.cc:686] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 LEADER]: Becoming Leader. State: Replica: 0530fe425ad54b7f83e44f18571ea8e2, State: Running, Role: LEADER
I20260926 05:51:54.180876 1163 consensus_queue.cc:237] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.181630 647 catalog_manager.cc:5495] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0530fe425ad54b7f83e44f18571ea8e2 (127.24.255.2). New cstate: current_term: 1 leader_uuid: "0530fe425ad54b7f83e44f18571ea8e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } health_report { overall_health: UNKNOWN } } }
Sep 26 05:51:54 dist-test-slave-7fr8 krb5kdc[602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401913, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.62@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:54.266332 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.258943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34585 (local address 127.0.0.1:57016)
0926 05:51:54.259143 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:54.259159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:54.259228 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:54.259354 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:54.259354 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:54.259700 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:54.259750 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.259752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.260457 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.260458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.261055 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.261058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.261430 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.261431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.261515 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.262019 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:54.262024 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:54.263398 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:54.265042 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:54.265050 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:54.265052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:54.265259 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:54.265559 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:54.265560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:54.265560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:54.265629 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:54.265955 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:54.265963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:54.266024 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 05:51:54.266128 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 05:51:54.266252 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
I20260926 05:51:54.266414 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.259074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.24.255.62:34585)
0926 05:51:54.259134 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:54.259136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:54.259230 (+ 94us) server_negotiation.cc:407] Connection header received
0926 05:51:54.259269 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:54.259269 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:54.259279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:54.259328 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:54.259848 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.260367 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.261146 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.261339 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.261456 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.263485 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:54.263487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:54.263487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:54.263494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:54.264944 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:54.265347 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:54.265348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:54.265348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:54.265408 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:54.265713 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:54.265714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:54.265715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:54.265847 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:54.265915 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:54.266245 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 05:51:54.266316 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:51:54.268916 647 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:57016:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 05:51:54.269114 647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:54.270517 924 tablet_service.cc:1467] Processing CreateTablet for tablet b251e53b197c435b8148786a6af11886 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c47627456f8d44c087aa7070a2325cc8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:51:54.270525 1064 tablet_service.cc:1467] Processing CreateTablet for tablet b251e53b197c435b8148786a6af11886 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c47627456f8d44c087aa7070a2325cc8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:51:54.270569 784 tablet_service.cc:1467] Processing CreateTablet for tablet b251e53b197c435b8148786a6af11886 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c47627456f8d44c087aa7070a2325cc8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:51:54.270685 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b251e53b197c435b8148786a6af11886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:54.270690 1064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b251e53b197c435b8148786a6af11886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:54.270706 784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b251e53b197c435b8148786a6af11886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:54.272063 1161 tablet_bootstrap.cc:492] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18: Bootstrap starting.
I20260926 05:51:54.272085 1158 tablet_bootstrap.cc:492] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1: Bootstrap starting.
I20260926 05:51:54.272387 1161 tablet_bootstrap.cc:654] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:54.272423 1158 tablet_bootstrap.cc:654] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:54.272713 1159 tablet_bootstrap.cc:492] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2: Bootstrap starting.
I20260926 05:51:54.272892 1161 tablet_bootstrap.cc:492] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18: No bootstrap required, opened a new log
I20260926 05:51:54.272996 1158 tablet_bootstrap.cc:492] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1: No bootstrap required, opened a new log
I20260926 05:51:54.273013 1161 ts_tablet_manager.cc:1397] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:54.273036 1158 ts_tablet_manager.cc:1397] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:54.273092 1159 tablet_bootstrap.cc:654] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:54.273168 1161 raft_consensus.cc:348] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.273171 1158 raft_consensus.cc:348] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.273226 1161 raft_consensus.cc:374] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:54.273242 1161 raft_consensus.cc:729] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f17271a0344c7d8b7d985759c31b18, State: Initialized, Role: FOLLOWER
I20260926 05:51:54.273241 1158 raft_consensus.cc:374] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:54.273264 1158 raft_consensus.cc:729] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8e33886c284db192cb2da2346880a1, State: Initialized, Role: FOLLOWER
I20260926 05:51:54.273290 1161 consensus_queue.cc:260] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.273307 1158 consensus_queue.cc:260] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.273391 1161 ts_tablet_manager.cc:1428] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 05:51:54.273407 1158 ts_tablet_manager.cc:1428] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:54.273444 1176 tablet_replica.cc:439] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } }
I20260926 05:51:54.273520 1176 tablet_replica.cc:442] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:54.273485 1164 tablet_replica.cc:439] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } }
I20260926 05:51:54.273561 1164 tablet_replica.cc:442] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:54.274492 1159 tablet_bootstrap.cc:492] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2: No bootstrap required, opened a new log
I20260926 05:51:54.274554 1159 ts_tablet_manager.cc:1397] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:51:54.274686 1159 raft_consensus.cc:348] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.274745 1159 raft_consensus.cc:374] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:54.274760 1159 raft_consensus.cc:729] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0530fe425ad54b7f83e44f18571ea8e2, State: Initialized, Role: FOLLOWER
I20260926 05:51:54.274806 1159 consensus_queue.cc:260] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.274907 1159 ts_tablet_manager.cc:1428] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:54.274964 1163 tablet_replica.cc:439] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } }
I20260926 05:51:54.275058 1163 tablet_replica.cc:442] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:54.338017 1164 raft_consensus.cc:482] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:54.338140 1164 raft_consensus.cc:504] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.338555 1164 leader_election.cc:288] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0530fe425ad54b7f83e44f18571ea8e2 (127.24.255.2:46623), 76f17271a0344c7d8b7d985759c31b18 (127.24.255.3:33373)
I20260926 05:51:54.342689 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.338832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:41097 (local address 127.24.255.2:46623)
0926 05:51:54.338874 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:54.338876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:54.339048 (+ 172us) server_negotiation.cc:407] Connection header received
0926 05:51:54.339088 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:54.339089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:54.339099 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:54.339167 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:54.339400 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.340176 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.341723 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.342339 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.342400 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.342407 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:54.342575 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 05:51:54.342619 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:51:54.342689 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.338852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:51947 (local address 127.24.255.3:33373)
0926 05:51:54.338928 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:54.338930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:54.339047 (+ 117us) server_negotiation.cc:407] Connection header received
0926 05:51:54.339090 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:54.339091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:54.339099 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:54.339167 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:51:54.339391 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.340156 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.341723 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.342307 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.342388 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.342399 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:54.342571 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 05:51:54.342615 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 05:51:54.342727 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.338744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:46623 (local address 127.24.255.1:41097)
0926 05:51:54.338935 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:54.338946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:54.339019 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:54.339178 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:54.339179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:54.339181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:54.339317 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.339319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.340248 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.340249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.341600 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.341605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.342457 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.342458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.342513 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.342515 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:51:54.342563 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:51:54.342604 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":58,"thread_start_us":118,"threads_started":1}
I20260926 05:51:54.342730 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.338753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:33373 (local address 127.24.255.1:51947)
0926 05:51:54.338937 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:54.338946 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:54.339019 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:54.339178 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:54.339179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:54.339182 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:51:54.339291 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.339294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.340246 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.340248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.341600 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.341605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.342389 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.342390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.342487 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.342493 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:51:54.342557 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:51:54.342604 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":34,"thread_start_us":107,"threads_started":1}
I20260926 05:51:54.343068 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b251e53b197c435b8148786a6af11886" candidate_uuid: "0b8e33886c284db192cb2da2346880a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f17271a0344c7d8b7d985759c31b18" is_pre_election: true
I20260926 05:51:54.343086 944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b251e53b197c435b8148786a6af11886" candidate_uuid: "0b8e33886c284db192cb2da2346880a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0530fe425ad54b7f83e44f18571ea8e2" is_pre_election: true
I20260926 05:51:54.343165 1084 raft_consensus.cc:2454] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b8e33886c284db192cb2da2346880a1 in term 0.
I20260926 05:51:54.343165 944 raft_consensus.cc:2454] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b8e33886c284db192cb2da2346880a1 in term 0.
I20260926 05:51:54.343326 736 leader_election.cc:302] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8e33886c284db192cb2da2346880a1, 76f17271a0344c7d8b7d985759c31b18; no voters:
I20260926 05:51:54.343495 1164 raft_consensus.cc:2781] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:54.343539 1164 raft_consensus.cc:482] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:54.343564 1164 raft_consensus.cc:3037] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:54.344242 1164 raft_consensus.cc:504] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.344359 1164 leader_election.cc:288] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [CANDIDATE]: Term 1 election: Requested vote from peers 0530fe425ad54b7f83e44f18571ea8e2 (127.24.255.2:46623), 76f17271a0344c7d8b7d985759c31b18 (127.24.255.3:33373)
I20260926 05:51:54.344571 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b251e53b197c435b8148786a6af11886" candidate_uuid: "0b8e33886c284db192cb2da2346880a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f17271a0344c7d8b7d985759c31b18"
I20260926 05:51:54.344578 944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b251e53b197c435b8148786a6af11886" candidate_uuid: "0b8e33886c284db192cb2da2346880a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0530fe425ad54b7f83e44f18571ea8e2"
I20260926 05:51:54.344645 944 raft_consensus.cc:3037] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:54.344648 1084 raft_consensus.cc:3037] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:54.345242 1084 raft_consensus.cc:2454] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b8e33886c284db192cb2da2346880a1 in term 1.
I20260926 05:51:54.345248 944 raft_consensus.cc:2454] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b8e33886c284db192cb2da2346880a1 in term 1.
I20260926 05:51:54.345410 737 leader_election.cc:302] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0530fe425ad54b7f83e44f18571ea8e2, 0b8e33886c284db192cb2da2346880a1; no voters:
I20260926 05:51:54.345505 1164 raft_consensus.cc:2781] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:54.345790 1164 raft_consensus.cc:686] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [term 1 LEADER]: Becoming Leader. State: Replica: 0b8e33886c284db192cb2da2346880a1, State: Running, Role: LEADER
I20260926 05:51:54.345929 1164 consensus_queue.cc:237] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.346417 1202 tablet_replica.cc:439] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1: TxnStatusTablet state changed. Reason: New leader 0b8e33886c284db192cb2da2346880a1. Latest consensus state: current_term: 1 leader_uuid: "0b8e33886c284db192cb2da2346880a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } }
I20260926 05:51:54.346506 1202 tablet_replica.cc:442] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:51:54.346750 647 catalog_manager.cc:5495] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b8e33886c284db192cb2da2346880a1 (127.24.255.1). New cstate: current_term: 1 leader_uuid: "0b8e33886c284db192cb2da2346880a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } health_report { overall_health: UNKNOWN } } }
I20260926 05:51:54.346952 1204 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 05:51:54.347095 1204 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 05:51:54.371258 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.367869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.24.255.1:45921)
0926 05:51:54.367923 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:54.367925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:54.368044 (+ 119us) server_negotiation.cc:407] Connection header received
0926 05:51:54.368131 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:54.368132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:54.368242 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:54.368357 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:54.368552 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.369081 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.370396 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.370605 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.370673 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.370884 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:54.371017 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:54.371084 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:54.371149 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 05:51:54.371174 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 05:51:54.371292 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.367796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:45921 (local address 127.0.0.1:47312)
0926 05:51:54.367948 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:54.367960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:54.368043 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:54.368374 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:54.368375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:54.368376 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:51:54.368449 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.368452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.369260 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.369260 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.370275 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.370278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.370685 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.370686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.370745 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.370768 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:51:54.371086 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:51:54.371093 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 05:51:54.371145 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 05:51:54.371205 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 05:51:54.372409 944 raft_consensus.cc:1261] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Refusing update from remote peer 0b8e33886c284db192cb2da2346880a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:54.372464 1084 raft_consensus.cc:1261] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Refusing update from remote peer 0b8e33886c284db192cb2da2346880a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:54.372675 1164 consensus_queue.cc:1035] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:54.372819 1202 consensus_queue.cc:1035] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:54.373373 1163 tablet_replica.cc:439] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2: TxnStatusTablet state changed. Reason: New leader 0b8e33886c284db192cb2da2346880a1. Latest consensus state: current_term: 1 leader_uuid: "0b8e33886c284db192cb2da2346880a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } }
I20260926 05:51:54.373492 1163 tablet_replica.cc:442] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:54.373476 1176 tablet_replica.cc:439] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18: TxnStatusTablet state changed. Reason: New leader 0b8e33886c284db192cb2da2346880a1. Latest consensus state: current_term: 1 leader_uuid: "0b8e33886c284db192cb2da2346880a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } }
I20260926 05:51:54.373553 1176 tablet_replica.cc:442] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:54.374207 1164 tablet_replica.cc:439] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b8e33886c284db192cb2da2346880a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } }
I20260926 05:51:54.374297 1164 tablet_replica.cc:442] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:51:54.374583 1172 mvcc.cc:205] Tried to move back new op lower bound from 7333486241267036160 to 7333486241161453568. Current Snapshot: MvccSnapshot[applied={T|T < 7333486241267036160 or (T in {7333486241267036160})}]
I20260926 05:51:54.374739 1164 tablet_replica.cc:439] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b8e33886c284db192cb2da2346880a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } }
I20260926 05:51:54.374804 1164 tablet_replica.cc:442] T b251e53b197c435b8148786a6af11886 P 0b8e33886c284db192cb2da2346880a1: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:51:54.374799 1173 mvcc.cc:205] Tried to move back new op lower bound from 7333486241267036160 to 7333486241161453568. Current Snapshot: MvccSnapshot[applied={T|T < 7333486241267036160}]
I20260926 05:51:54.374902 1163 tablet_replica.cc:439] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b8e33886c284db192cb2da2346880a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } }
I20260926 05:51:54.374966 1163 tablet_replica.cc:442] T b251e53b197c435b8148786a6af11886 P 0530fe425ad54b7f83e44f18571ea8e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:54.375869 1176 tablet_replica.cc:439] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b8e33886c284db192cb2da2346880a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } } }
I20260926 05:51:54.375963 1176 tablet_replica.cc:442] T b251e53b197c435b8148786a6af11886 P 76f17271a0344c7d8b7d985759c31b18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:51:54.378793 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.375982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.24.255.2:46623)
0926 05:51:54.376017 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:54.376019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:54.376036 (+ 17us) server_negotiation.cc:407] Connection header received
0926 05:51:54.376073 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:54.376074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:54.376094 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:54.376147 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:54.376301 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.377036 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.377855 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.378187 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.378255 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.378406 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:54.378542 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:54.378615 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:54.378692 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 05:51:54.378712 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:51:54.384353 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.381507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:41655 (local address 127.24.255.1:45921)
0926 05:51:54.381590 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:54.381592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:54.381737 (+ 145us) server_negotiation.cc:407] Connection header received
0926 05:51:54.381828 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:54.381829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:54.381838 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:54.381879 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:54.382006 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.382560 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.383695 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.383885 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.383940 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.384140 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:54.384171 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:54.384219 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:54.384272 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 05:51:54.384292 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 05:51:54.384462 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.381440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:45921 (local address 127.24.255.2:41655)
0926 05:51:54.381666 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:54.381677 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:54.381741 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:54.381890 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:54.381891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:54.381892 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:51:54.381951 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.381952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.382628 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.382629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.383605 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.383607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.383961 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.383962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.384028 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.384047 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:51:54.384229 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:51:54.384230 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:51:54.384271 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 05:51:54.384315 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
I20260926 05:51:54.386912 804 raft_consensus.cc:1261] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Refusing update from remote peer 0530fe425ad54b7f83e44f18571ea8e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:54.386938 1084 raft_consensus.cc:1261] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Refusing update from remote peer 0530fe425ad54b7f83e44f18571ea8e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:54.387293 1189 consensus_queue.cc:1035] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:54.387434 1163 consensus_queue.cc:1035] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:54.397497 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.394241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:39487 (local address 127.24.255.2:46623)
0926 05:51:54.394300 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:54.394302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:54.394379 (+ 77us) server_negotiation.cc:407] Connection header received
0926 05:51:54.394496 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:54.394498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:54.394519 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:54.394616 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:54.394944 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.395527 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.396530 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.396823 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.396886 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.397197 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:54.397241 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:54.397315 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:54.397381 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 05:51:54.397412 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 05:51:54.397543 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.393825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:46623 (local address 127.24.255.1:39487)
0926 05:51:54.394248 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:54.394259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:54.394303 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:54.394705 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:54.394706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:54.394709 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:51:54.394807 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.394810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.395624 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.395625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.396397 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:54.396444 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.396906 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.396907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:54.397030 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.397051 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:51:54.397323 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:51:54.397325 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:51:54.397380 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 05:51:54.397433 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":77,"threads_started":1}
I20260926 05:51:54.408288 646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:54.408424 646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:54.409097 646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993: Sending DeleteTablet for 3 replicas of tablet cfad9485fba94f56be24a4619ab371ee
I20260926 05:51:54.409411 1064 tablet_service.cc:1514] Processing DeleteTablet for tablet cfad9485fba94f56be24a4619ab371ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:54 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:58080
I20260926 05:51:54.409404 924 tablet_service.cc:1514] Processing DeleteTablet for tablet cfad9485fba94f56be24a4619ab371ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:54 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:56588
I20260926 05:51:54.409509 1182 tablet_replica.cc:330] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18: stopping tablet replica
I20260926 05:51:54.409521 1183 tablet_replica.cc:330] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2: stopping tablet replica
I20260926 05:51:54.409567 1182 raft_consensus.cc:2229] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:54.409574 1183 raft_consensus.cc:2229] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:51:54.409610 1182 raft_consensus.cc:2258] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:54.409670 1183 raft_consensus.cc:2258] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:54.410322 1183 ts_tablet_manager.cc:1905] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:54.410496 1182 ts_tablet_manager.cc:1905] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:54.412133 1182 ts_tablet_manager.cc:1918] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:54.412132 1183 ts_tablet_manager.cc:1918] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:54.412197 1182 log.cc:1196] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/cfad9485fba94f56be24a4619ab371ee
I20260926 05:51:54.412212 1183 log.cc:1196] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/cfad9485fba94f56be24a4619ab371ee
I20260926 05:51:54.412418 1182 ts_tablet_manager.cc:1939] T cfad9485fba94f56be24a4619ab371ee P 76f17271a0344c7d8b7d985759c31b18: Deleting consensus metadata
I20260926 05:51:54.412426 1183 ts_tablet_manager.cc:1939] T cfad9485fba94f56be24a4619ab371ee P 0530fe425ad54b7f83e44f18571ea8e2: Deleting consensus metadata
I20260926 05:51:54.412602 784 tablet_service.cc:1514] Processing DeleteTablet for tablet cfad9485fba94f56be24a4619ab371ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:54 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:47300
I20260926 05:51:54.412700 1181 tablet_replica.cc:330] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1: stopping tablet replica
I20260926 05:51:54.412743 1181 raft_consensus.cc:2229] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:54.412778 1181 raft_consensus.cc:2258] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:54.412900 633 catalog_manager.cc:4841] TS 0530fe425ad54b7f83e44f18571ea8e2 (127.24.255.2:46623): tablet cfad9485fba94f56be24a4619ab371ee (table test-table [id=77f169915134491dba51acb841975e35]) successfully deleted
I20260926 05:51:54.413053 632 catalog_manager.cc:4841] TS 76f17271a0344c7d8b7d985759c31b18 (127.24.255.3:33373): tablet cfad9485fba94f56be24a4619ab371ee (table test-table [id=77f169915134491dba51acb841975e35]) successfully deleted
I20260926 05:51:54.413714 1181 ts_tablet_manager.cc:1905] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:54.415549 1181 ts_tablet_manager.cc:1918] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:54.415659 1181 log.cc:1196] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/wal/wals/cfad9485fba94f56be24a4619ab371ee
I20260926 05:51:54.415864 1181 ts_tablet_manager.cc:1939] T cfad9485fba94f56be24a4619ab371ee P 0b8e33886c284db192cb2da2346880a1: Deleting consensus metadata
I20260926 05:51:54.416661 632 catalog_manager.cc:4841] TS 0b8e33886c284db192cb2da2346880a1 (127.24.255.1:45921): tablet cfad9485fba94f56be24a4619ab371ee (table test-table [id=77f169915134491dba51acb841975e35]) successfully deleted
Sep 26 05:51:54 dist-test-slave-7fr8 krb5kdc[602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:51:54 dist-test-slave-7fr8 krb5kdc[602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:54 dist-test-slave-7fr8 krb5kdc[602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 05:51:54.436695 704 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:54.431589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.24.255.62:34585)
0926 05:51:54.431648 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:54.431650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:54.431745 (+ 95us) server_negotiation.cc:407] Connection header received
0926 05:51:54.431810 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:54.431811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:54.431821 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:54.431868 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:54.432434 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.432923 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.433622 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.433824 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.433877 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.436623 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57024: BlockingRecv error: recv got EOF from 127.0.0.1:57024 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 05:51:54 dist-test-slave-7fr8 krb5kdc[602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM
I20260926 05:51:54.445559 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.437454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.24.255.62:34585)
0926 05:51:54.437506 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:54.437507 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:54.437751 (+ 244us) server_negotiation.cc:407] Connection header received
0926 05:51:54.437810 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:54.437811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:54.437821 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:54.437867 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:54.438402 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.438988 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.439747 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.440126 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.440357 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.443230 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:54.443232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:54.443232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:54.443241 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:54.444529 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:54.444847 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:54.444848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:54.444849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:54.444914 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:54.445080 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:54.445081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:54.445081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:54.445197 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:54.445283 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:54.445381 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 05:51:54.445465 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:51:54.447149 646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:51:54.447314 646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:54.448649 784 tablet_service.cc:1467] Processing CreateTablet for tablet 280cf2062bf94cb1a7abf645784e23ae (DEFAULT_TABLE table=test-table [id=3cdaaaa891bc4436a54f088242abc0e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:54.448753 924 tablet_service.cc:1467] Processing CreateTablet for tablet 280cf2062bf94cb1a7abf645784e23ae (DEFAULT_TABLE table=test-table [id=3cdaaaa891bc4436a54f088242abc0e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:54.448807 784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280cf2062bf94cb1a7abf645784e23ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:54.448851 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280cf2062bf94cb1a7abf645784e23ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:54.450258 1064 tablet_service.cc:1467] Processing CreateTablet for tablet 280cf2062bf94cb1a7abf645784e23ae (DEFAULT_TABLE table=test-table [id=3cdaaaa891bc4436a54f088242abc0e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:54.450322 1158 tablet_bootstrap.cc:492] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1: Bootstrap starting.
I20260926 05:51:54.450376 1064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 280cf2062bf94cb1a7abf645784e23ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:54.450722 1158 tablet_bootstrap.cc:654] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:54.451589 1161 tablet_bootstrap.cc:492] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18: Bootstrap starting.
I20260926 05:51:54.451916 1161 tablet_bootstrap.cc:654] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:54.452121 1159 tablet_bootstrap.cc:492] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2: Bootstrap starting.
I20260926 05:51:54.452509 1159 tablet_bootstrap.cc:654] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:54.453085 1159 tablet_bootstrap.cc:492] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2: No bootstrap required, opened a new log
I20260926 05:51:54.453135 1159 ts_tablet_manager.cc:1397] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:54.453285 1159 raft_consensus.cc:348] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.453356 1159 raft_consensus.cc:374] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:54.453374 1159 raft_consensus.cc:729] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0530fe425ad54b7f83e44f18571ea8e2, State: Initialized, Role: FOLLOWER
I20260926 05:51:54.453414 1159 consensus_queue.cc:260] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.453434 1158 tablet_bootstrap.cc:492] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1: No bootstrap required, opened a new log
I20260926 05:51:54.453488 1158 ts_tablet_manager.cc:1397] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 05:51:54.453498 1159 ts_tablet_manager.cc:1428] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:54.453644 1158 raft_consensus.cc:348] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.453722 1158 raft_consensus.cc:374] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:54.453740 1158 raft_consensus.cc:729] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8e33886c284db192cb2da2346880a1, State: Initialized, Role: FOLLOWER
I20260926 05:51:54.453774 1158 consensus_queue.cc:260] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.453843 1158 ts_tablet_manager.cc:1428] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:51:54.454629 1161 tablet_bootstrap.cc:492] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18: No bootstrap required, opened a new log
I20260926 05:51:54.454679 1161 ts_tablet_manager.cc:1397] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 05:51:54.454785 1161 raft_consensus.cc:348] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.454844 1161 raft_consensus.cc:374] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:54.454861 1161 raft_consensus.cc:729] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f17271a0344c7d8b7d985759c31b18, State: Initialized, Role: FOLLOWER
I20260926 05:51:54.454903 1161 consensus_queue.cc:260] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.454985 1161 ts_tablet_manager.cc:1428] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:54.501955 1176 raft_consensus.cc:482] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:54.502084 1176 raft_consensus.cc:504] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.502285 1176 leader_election.cc:288] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b8e33886c284db192cb2da2346880a1 (127.24.255.1:45921), 0530fe425ad54b7f83e44f18571ea8e2 (127.24.255.2:46623)
I20260926 05:51:54.502522 804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280cf2062bf94cb1a7abf645784e23ae" candidate_uuid: "76f17271a0344c7d8b7d985759c31b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8e33886c284db192cb2da2346880a1" is_pre_election: true
I20260926 05:51:54.502604 804 raft_consensus.cc:2454] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76f17271a0344c7d8b7d985759c31b18 in term 0.
I20260926 05:51:54.502702 944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280cf2062bf94cb1a7abf645784e23ae" candidate_uuid: "76f17271a0344c7d8b7d985759c31b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0530fe425ad54b7f83e44f18571ea8e2" is_pre_election: true
I20260926 05:51:54.502779 944 raft_consensus.cc:2454] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76f17271a0344c7d8b7d985759c31b18 in term 0.
I20260926 05:51:54.502789 1016 leader_election.cc:302] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8e33886c284db192cb2da2346880a1, 76f17271a0344c7d8b7d985759c31b18; no voters:
I20260926 05:51:54.503049 1176 raft_consensus.cc:2781] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:54.503093 1176 raft_consensus.cc:482] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:54.503111 1176 raft_consensus.cc:3037] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:54.504021 1176 raft_consensus.cc:504] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.504208 1176 leader_election.cc:288] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [CANDIDATE]: Term 1 election: Requested vote from peers 0b8e33886c284db192cb2da2346880a1 (127.24.255.1:45921), 0530fe425ad54b7f83e44f18571ea8e2 (127.24.255.2:46623)
I20260926 05:51:54.504460 804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280cf2062bf94cb1a7abf645784e23ae" candidate_uuid: "76f17271a0344c7d8b7d985759c31b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8e33886c284db192cb2da2346880a1"
I20260926 05:51:54.504532 804 raft_consensus.cc:3037] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:54.505364 804 raft_consensus.cc:2454] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76f17271a0344c7d8b7d985759c31b18 in term 1.
I20260926 05:51:54.505477 944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "280cf2062bf94cb1a7abf645784e23ae" candidate_uuid: "76f17271a0344c7d8b7d985759c31b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0530fe425ad54b7f83e44f18571ea8e2"
I20260926 05:51:54.505553 944 raft_consensus.cc:3037] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:54.505574 1016 leader_election.cc:302] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8e33886c284db192cb2da2346880a1, 76f17271a0344c7d8b7d985759c31b18; no voters:
I20260926 05:51:54.505883 1176 raft_consensus.cc:2781] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:54.505990 1176 raft_consensus.cc:686] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [term 1 LEADER]: Becoming Leader. State: Replica: 76f17271a0344c7d8b7d985759c31b18, State: Running, Role: LEADER
I20260926 05:51:54.506042 1176 consensus_queue.cc:237] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.506296 944 raft_consensus.cc:2454] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76f17271a0344c7d8b7d985759c31b18 in term 1.
I20260926 05:51:54.506647 646 catalog_manager.cc:5495] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76f17271a0344c7d8b7d985759c31b18 (127.24.255.3). New cstate: current_term: 1 leader_uuid: "76f17271a0344c7d8b7d985759c31b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } health_report { overall_health: HEALTHY } } }
I20260926 05:51:54.518191 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.515186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.24.255.3:33373)
0926 05:51:54.515230 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:54.515232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:54.515242 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:54.515309 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:54.515310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:54.515321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:54.515388 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:54.515530 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.516289 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.517404 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.517700 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.517779 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.517917 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:54.517954 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:54.518001 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:54.518056 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 05:51:54.518080 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:51:54.518880 944 raft_consensus.cc:1261] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Refusing update from remote peer 76f17271a0344c7d8b7d985759c31b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:54.519093 804 raft_consensus.cc:1261] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Refusing update from remote peer 76f17271a0344c7d8b7d985759c31b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:54.519323 1176 consensus_queue.cc:1035] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:54.519460 1176 consensus_queue.cc:1035] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:54.522550 647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:54.522652 647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:54.523552 647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993: Sending DeleteTablet for 3 replicas of tablet 280cf2062bf94cb1a7abf645784e23ae
I20260926 05:51:54.523799 924 tablet_service.cc:1514] Processing DeleteTablet for tablet 280cf2062bf94cb1a7abf645784e23ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:54 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:56588
I20260926 05:51:54.523955 1183 tablet_replica.cc:330] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2: stopping tablet replica
I20260926 05:51:54.524011 1183 raft_consensus.cc:2229] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:54.523989 784 tablet_service.cc:1514] Processing DeleteTablet for tablet 280cf2062bf94cb1a7abf645784e23ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:54 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:47300
I20260926 05:51:54.524055 1183 raft_consensus.cc:2258] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:54.524085 1064 tablet_service.cc:1514] Processing DeleteTablet for tablet 280cf2062bf94cb1a7abf645784e23ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:54 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:58080
I20260926 05:51:54.524166 1181 tablet_replica.cc:330] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1: stopping tablet replica
I20260926 05:51:54.524217 1181 raft_consensus.cc:2229] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:54.524250 1181 pending_rounds.cc:62] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1: Trying to abort 2 pending ops.
I20260926 05:51:54.524266 1181 pending_rounds.cc:68] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333486241867771904 op_type: WRITE_OP write_request { tablet_id: "280cf2062bf94cb1a7abf645784e23ae" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\246\275\335\325\006\"8\n\ntest-admin\022*\n 3cdaaaa891bc4436a54f088242abc0e3\020\001\030\001 \001(\001" signature: "<redacted>""K\311\3553\235\305PmY\037\205\376\265.f\323\345,^\316\014\323\312\263\003\231u\260\323\305d\255\375\367j\tr0\352s\\=G\320\226\373\026\003\241\277\236\037\215pk\350\330m\374\202|\332)\215" signing_key_seq_num: 0 } } request_id { client_id: "0397aa99b6f042a9a5de048b3bfea4fa" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260926 05:51:54.524438 1181 pending_rounds.cc:68] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333486241817149440 op_type: NO_OP noop_request { }
I20260926 05:51:54.524386 647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 05:51:54.524479 1181 raft_consensus.cc:2866] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 05:51:54.524508 1181 raft_consensus.cc:2258] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 05:51:54.524526 647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:51:54.524576 1182 tablet_replica.cc:330] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18: stopping tablet replica
I20260926 05:51:54.524631 1182 raft_consensus.cc:2229] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:51:54.524691 1182 raft_consensus.cc:2258] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:54.524850 1183 ts_tablet_manager.cc:1905] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:54.525895 1182 ts_tablet_manager.cc:1905] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:54.526427 1183 ts_tablet_manager.cc:1918] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:54.526471 1183 log.cc:1196] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/280cf2062bf94cb1a7abf645784e23ae
I20260926 05:51:54.526618 1183 ts_tablet_manager.cc:1939] T 280cf2062bf94cb1a7abf645784e23ae P 0530fe425ad54b7f83e44f18571ea8e2: Deleting consensus metadata
I20260926 05:51:54.527254 633 catalog_manager.cc:4841] TS 0530fe425ad54b7f83e44f18571ea8e2 (127.24.255.2:46623): tablet 280cf2062bf94cb1a7abf645784e23ae (table test-table [id=3cdaaaa891bc4436a54f088242abc0e3]) successfully deleted
I20260926 05:51:54.529353 924 tablet_service.cc:1467] Processing CreateTablet for tablet 67ed873c43a14707a65594024f57e057 (DEFAULT_TABLE table=test-table [id=3b41b1d3faa44fecb9dca1372aecae0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:54.529490 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ed873c43a14707a65594024f57e057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:54.530838 1159 tablet_bootstrap.cc:492] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2: Bootstrap starting.
I20260926 05:51:54.531245 1159 tablet_bootstrap.cc:654] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:54.531770 1064 tablet_service.cc:1467] Processing CreateTablet for tablet 67ed873c43a14707a65594024f57e057 (DEFAULT_TABLE table=test-table [id=3b41b1d3faa44fecb9dca1372aecae0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:54.531939 784 tablet_service.cc:1467] Processing CreateTablet for tablet 67ed873c43a14707a65594024f57e057 (DEFAULT_TABLE table=test-table [id=3b41b1d3faa44fecb9dca1372aecae0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:51:54.532047 784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ed873c43a14707a65594024f57e057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:54.533418 1158 tablet_bootstrap.cc:492] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1: Bootstrap starting.
I20260926 05:51:54.533753 1158 tablet_bootstrap.cc:654] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:54.534507 1181 ts_tablet_manager.cc:1905] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:54.534755 1158 tablet_bootstrap.cc:492] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1: No bootstrap required, opened a new log
I20260926 05:51:54.534811 1158 ts_tablet_manager.cc:1397] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:54.534945 1158 raft_consensus.cc:348] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.535019 1158 raft_consensus.cc:374] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:54.535038 1158 raft_consensus.cc:729] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8e33886c284db192cb2da2346880a1, State: Initialized, Role: FOLLOWER
I20260926 05:51:54.535089 1158 consensus_queue.cc:260] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.535187 1158 ts_tablet_manager.cc:1428] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:54.536144 1159 tablet_bootstrap.cc:492] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2: No bootstrap required, opened a new log
I20260926 05:51:54.536203 1159 ts_tablet_manager.cc:1397] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260926 05:51:54.536362 1159 raft_consensus.cc:348] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.536439 1159 raft_consensus.cc:374] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:54.536458 1159 raft_consensus.cc:729] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0530fe425ad54b7f83e44f18571ea8e2, State: Initialized, Role: FOLLOWER
I20260926 05:51:54.536501 1159 consensus_queue.cc:260] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.536597 1159 ts_tablet_manager.cc:1428] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:54.536790 1181 ts_tablet_manager.cc:1918] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:54.536863 1181 log.cc:1196] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-0/wal/wals/280cf2062bf94cb1a7abf645784e23ae
I20260926 05:51:54.537055 1181 ts_tablet_manager.cc:1939] T 280cf2062bf94cb1a7abf645784e23ae P 0b8e33886c284db192cb2da2346880a1: Deleting consensus metadata
I20260926 05:51:54.537384 1182 ts_tablet_manager.cc:1918] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:51:54.537392 632 catalog_manager.cc:4841] TS 0b8e33886c284db192cb2da2346880a1 (127.24.255.1:45921): tablet 280cf2062bf94cb1a7abf645784e23ae (table test-table [id=3cdaaaa891bc4436a54f088242abc0e3]) successfully deleted
I20260926 05:51:54.537426 1064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ed873c43a14707a65594024f57e057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:54.537446 1182 log.cc:1196] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/280cf2062bf94cb1a7abf645784e23ae
I20260926 05:51:54.537636 1182 ts_tablet_manager.cc:1939] T 280cf2062bf94cb1a7abf645784e23ae P 76f17271a0344c7d8b7d985759c31b18: Deleting consensus metadata
I20260926 05:51:54.537957 632 catalog_manager.cc:4841] TS 76f17271a0344c7d8b7d985759c31b18 (127.24.255.3:33373): tablet 280cf2062bf94cb1a7abf645784e23ae (table test-table [id=3cdaaaa891bc4436a54f088242abc0e3]) successfully deleted
I20260926 05:51:54.538780 1161 tablet_bootstrap.cc:492] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18: Bootstrap starting.
I20260926 05:51:54.539177 1161 tablet_bootstrap.cc:654] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:54.539691 1161 tablet_bootstrap.cc:492] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18: No bootstrap required, opened a new log
I20260926 05:51:54.539750 1161 ts_tablet_manager.cc:1397] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:54.539857 1161 raft_consensus.cc:348] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.539924 1161 raft_consensus.cc:374] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:54.539942 1161 raft_consensus.cc:729] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f17271a0344c7d8b7d985759c31b18, State: Initialized, Role: FOLLOWER
I20260926 05:51:54.539984 1161 consensus_queue.cc:260] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.540073 1161 ts_tablet_manager.cc:1428] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:54.637172 1163 raft_consensus.cc:482] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:51:54.637272 1163 raft_consensus.cc:504] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.637601 1163 leader_election.cc:288] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b8e33886c284db192cb2da2346880a1 (127.24.255.1:45921), 76f17271a0344c7d8b7d985759c31b18 (127.24.255.3:33373)
I20260926 05:51:54.637738 804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ed873c43a14707a65594024f57e057" candidate_uuid: "0530fe425ad54b7f83e44f18571ea8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8e33886c284db192cb2da2346880a1" is_pre_election: true
I20260926 05:51:54.637843 804 raft_consensus.cc:2454] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0530fe425ad54b7f83e44f18571ea8e2 in term 0.
I20260926 05:51:54.637907 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ed873c43a14707a65594024f57e057" candidate_uuid: "0530fe425ad54b7f83e44f18571ea8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f17271a0344c7d8b7d985759c31b18" is_pre_election: true
I20260926 05:51:54.638019 1084 raft_consensus.cc:2454] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0530fe425ad54b7f83e44f18571ea8e2 in term 0.
I20260926 05:51:54.638017 876 leader_election.cc:302] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0530fe425ad54b7f83e44f18571ea8e2, 0b8e33886c284db192cb2da2346880a1; no voters:
I20260926 05:51:54.638142 1163 raft_consensus.cc:2781] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:51:54.638193 1163 raft_consensus.cc:482] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:51:54.638213 1163 raft_consensus.cc:3037] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:54.638922 1163 raft_consensus.cc:504] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.639047 1163 leader_election.cc:288] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [CANDIDATE]: Term 1 election: Requested vote from peers 0b8e33886c284db192cb2da2346880a1 (127.24.255.1:45921), 76f17271a0344c7d8b7d985759c31b18 (127.24.255.3:33373)
I20260926 05:51:54.639236 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ed873c43a14707a65594024f57e057" candidate_uuid: "0530fe425ad54b7f83e44f18571ea8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f17271a0344c7d8b7d985759c31b18"
I20260926 05:51:54.639237 804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ed873c43a14707a65594024f57e057" candidate_uuid: "0530fe425ad54b7f83e44f18571ea8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8e33886c284db192cb2da2346880a1"
I20260926 05:51:54.639314 1084 raft_consensus.cc:3037] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:54.639315 804 raft_consensus.cc:3037] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:54.640069 1084 raft_consensus.cc:2454] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0530fe425ad54b7f83e44f18571ea8e2 in term 1.
I20260926 05:51:54.640069 804 raft_consensus.cc:2454] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0530fe425ad54b7f83e44f18571ea8e2 in term 1.
I20260926 05:51:54.640230 876 leader_election.cc:302] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0530fe425ad54b7f83e44f18571ea8e2, 0b8e33886c284db192cb2da2346880a1; no voters:
I20260926 05:51:54.640327 1163 raft_consensus.cc:2781] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:54.640378 1163 raft_consensus.cc:686] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 LEADER]: Becoming Leader. State: Replica: 0530fe425ad54b7f83e44f18571ea8e2, State: Running, Role: LEADER
I20260926 05:51:54.640425 1163 consensus_queue.cc:237] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } }
I20260926 05:51:54.641001 647 catalog_manager.cc:5495] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0530fe425ad54b7f83e44f18571ea8e2 (127.24.255.2). New cstate: current_term: 1 leader_uuid: "0530fe425ad54b7f83e44f18571ea8e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0530fe425ad54b7f83e44f18571ea8e2" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 } health_report { overall_health: UNKNOWN } } }
I20260926 05:51:54.664263 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.661939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.24.255.2:46623)
0926 05:51:54.662030 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:54.662032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:54.662040 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:54.662060 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:54.662060 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:54.662071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:54.662149 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:54.662268 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.662885 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.663617 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:54.663809 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:54.663863 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:54.664030 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:54.664065 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:54.664105 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:54.664160 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 05:51:54.664182 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 05:51:54.666977 1084 raft_consensus.cc:1261] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Refusing update from remote peer 0530fe425ad54b7f83e44f18571ea8e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:54.666986 804 raft_consensus.cc:1261] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Refusing update from remote peer 0530fe425ad54b7f83e44f18571ea8e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:54.667236 1215 consensus_queue.cc:1035] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b8e33886c284db192cb2da2346880a1" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 45921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:54.667361 1215 consensus_queue.cc:1035] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76f17271a0344c7d8b7d985759c31b18" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:54.678155 647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:54.678280 647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:51:54.679097 647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ee95393788043e58ef4623c85fa2993: Sending DeleteTablet for 3 replicas of tablet 67ed873c43a14707a65594024f57e057
I20260926 05:51:54.679342 784 tablet_service.cc:1514] Processing DeleteTablet for tablet 67ed873c43a14707a65594024f57e057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:54 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:47300
I20260926 05:51:54.679337 924 tablet_service.cc:1514] Processing DeleteTablet for tablet 67ed873c43a14707a65594024f57e057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:54 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:56588
I20260926 05:51:54.679533 1064 tablet_service.cc:1514] Processing DeleteTablet for tablet 67ed873c43a14707a65594024f57e057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:51:54 UTC) from {username='oryx', principal='oryx/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:58080
I20260926 05:51:54.679557 1183 tablet_replica.cc:330] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2: stopping tablet replica
I20260926 05:51:54.679781 1182 tablet_replica.cc:330] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18: stopping tablet replica
I20260926 05:51:54.679792 1183 raft_consensus.cc:2229] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:51:54.679859 1182 raft_consensus.cc:2229] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:54.679880 1183 raft_consensus.cc:2258] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:54.679981 1182 raft_consensus.cc:2258] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:54.680079 1181 tablet_replica.cc:330] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1: stopping tablet replica
I20260926 05:51:54.680142 1181 raft_consensus.cc:2229] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:51:54.680173 1181 raft_consensus.cc:2258] T 67ed873c43a14707a65594024f57e057 P 0b8e33886c284db192cb2da2346880a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:51:54.680171 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 719
I20260926 05:51:54.680357 1182 ts_tablet_manager.cc:1905] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:54.681241 1183 ts_tablet_manager.cc:1905] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:51:54.682749 1182 ts_tablet_manager.cc:1918] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:54.682760 1183 ts_tablet_manager.cc:1918] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:51:54.682806 1182 log.cc:1196] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/67ed873c43a14707a65594024f57e057
I20260926 05:51:54.682811 1183 log.cc:1196] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/67ed873c43a14707a65594024f57e057
I20260926 05:51:54.683017 1183 ts_tablet_manager.cc:1939] T 67ed873c43a14707a65594024f57e057 P 0530fe425ad54b7f83e44f18571ea8e2: Deleting consensus metadata
I20260926 05:51:54.683030 1182 ts_tablet_manager.cc:1939] T 67ed873c43a14707a65594024f57e057 P 76f17271a0344c7d8b7d985759c31b18: Deleting consensus metadata
I20260926 05:51:54.683355 633 catalog_manager.cc:4841] TS 0530fe425ad54b7f83e44f18571ea8e2 (127.24.255.2:46623): tablet 67ed873c43a14707a65594024f57e057 (table test-table [id=3b41b1d3faa44fecb9dca1372aecae0c]) successfully deleted
I20260926 05:51:54.683429 632 catalog_manager.cc:4841] TS 76f17271a0344c7d8b7d985759c31b18 (127.24.255.3:33373): tablet 67ed873c43a14707a65594024f57e057 (table test-table [id=3b41b1d3faa44fecb9dca1372aecae0c]) successfully deleted
W20260926 05:51:54.688246 632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:45921 (error 108)
I20260926 05:51:54.688467 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 859
I20260926 05:51:54.689384 1249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:54.689191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:45921 (local address 127.0.0.1:47326)
0926 05:51:54.689290 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:54.689337 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":47,"threads_started":1}
W20260926 05:51:54.690176 632 catalog_manager.cc:4584] TS 0b8e33886c284db192cb2da2346880a1 (127.24.255.1:45921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67ed873c43a14707a65594024f57e057: Network error: Client connection negotiation failed: client connection to 127.24.255.1:45921: connect: Connection refused (error 111)
I20260926 05:51:54.694437 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 999
I20260926 05:51:54.699579 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 615
2026-09-26T05:51:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2728 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:54 dist-test-slave-7fr8 krb5kdc[1252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:54 dist-test-slave-7fr8 krb5kdc[1252](info): set up 2 sockets
Sep 26 05:51:54 dist-test-slave-7fr8 krb5kdc[1252](info): commencing operation
krb5kdc: starting...
W20260926 05:51:55.678723 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:55 dist-test-slave-7fr8 krb5kdc[1252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:51:55.801285 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:40951
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:40951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:55.882303 1265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:55.882537 1265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:55.882561 1265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:55.883736 1265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:55.883782 1265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:55.883796 1265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:55.883807 1265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:55.883818 1265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:55.885324 1265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:40951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:40951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:55.885555 1265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:55.885773 1265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:55.890769 1271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:55.890959 1265 server_base.cc:1034] running on GCE node
W20260926 05:51:55.890774 1273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:55.890956 1270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:55.891506 1265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:55.891801 1265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:55.892957 1265 hybrid_clock.cc:648] HybridClock initialized: now 1790401915892942 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T05:51:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1198 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:55 dist-test-slave-7fr8 krb5kdc[1278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:55 dist-test-slave-7fr8 krb5kdc[1278](info): set up 2 sockets
Sep 26 05:51:55 dist-test-slave-7fr8 krb5kdc[1278](info): commencing operation
krb5kdc: starting...
W20260926 05:51:56.878772 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:51:56 dist-test-slave-7fr8 krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:51:57.005259 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:34919
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:34919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:57.089794 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:57.090049 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:57.090085 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:57.091332 1291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:57.091387 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:57.091405 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:57.091423 1291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:57.091437 1291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:57.093034 1291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:34919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:34919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:57.093312 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:57.093537 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:51:57.098392 1291 server_base.cc:1034] running on GCE node
W20260926 05:51:57.098295 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:57.098330 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:57.098295 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:57.098810 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:57.099099 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:57.100244 1291 hybrid_clock.cc:648] HybridClock initialized: now 1790401917100232 us; error 31 us; skew 500 ppm
Sep 26 05:51:57 dist-test-slave-7fr8 krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:57.109226 1291 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:51:57.109836 1291 webserver.cc:460] Webserver started at http://127.24.255.62:40733/ using document root <none> and password file <none>
I20260926 05:51:57.110087 1291 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:57.110177 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:57.110314 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:57.111380 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "36443d31b41e498fbe508353fdd8b3d8"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "296217833ed8e172cc054abb1674b66c"
server_key_iv: "aa0b5256c31a2d7d6c37a699d02e25fb"
server_key_version: "encryptionkey@0"
I20260926 05:51:57.111732 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "36443d31b41e498fbe508353fdd8b3d8"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "296217833ed8e172cc054abb1674b66c"
server_key_iv: "aa0b5256c31a2d7d6c37a699d02e25fb"
server_key_version: "encryptionkey@0"
I20260926 05:51:57.113107 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:51:57.114043 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:57.114259 1291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:57.114348 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "36443d31b41e498fbe508353fdd8b3d8"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "296217833ed8e172cc054abb1674b66c"
server_key_iv: "aa0b5256c31a2d7d6c37a699d02e25fb"
server_key_version: "encryptionkey@0"
I20260926 05:51:57.114423 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:57.122853 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:57.125118 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:57.125262 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:57.129379 1291 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:34919
I20260926 05:51:57.129469 1358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:34919 every 8 connection(s)
I20260926 05:51:57.129979 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:57.130342 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 1291
I20260926 05:51:57.130447 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:51:57.130666 25596 external_mini_cluster.cc:1428] Setting key 03483da914f2cb58e62f60913c5e9c46
I20260926 05:51:57.130885 1359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:57.133731 1359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8: Bootstrap starting.
I20260926 05:51:57.134395 1359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:57.134750 1359 log.cc:824] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:57.135666 1359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8: No bootstrap required, opened a new log
Sep 26 05:51:57 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:57.138445 1359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36443d31b41e498fbe508353fdd8b3d8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34919 } }
I20260926 05:51:57.138653 1359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:57.138695 1359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36443d31b41e498fbe508353fdd8b3d8, State: Initialized, Role: FOLLOWER
I20260926 05:51:57.138825 1359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36443d31b41e498fbe508353fdd8b3d8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34919 } }
I20260926 05:51:57.138906 1359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:57.138938 1359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:57.139082 1359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:57.139919 1359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36443d31b41e498fbe508353fdd8b3d8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34919 } }
I20260926 05:51:57.140108 1359 leader_election.cc:302] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36443d31b41e498fbe508353fdd8b3d8; no voters:
I20260926 05:51:57.140318 1359 leader_election.cc:288] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:57.140376 1364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:57.140532 1364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 1 LEADER]: Becoming Leader. State: Replica: 36443d31b41e498fbe508353fdd8b3d8, State: Running, Role: LEADER
I20260926 05:51:57.140604 1359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:57.140661 1364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36443d31b41e498fbe508353fdd8b3d8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34919 } }
I20260926 05:51:57.141147 1366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36443d31b41e498fbe508353fdd8b3d8. Latest consensus state: current_term: 1 leader_uuid: "36443d31b41e498fbe508353fdd8b3d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36443d31b41e498fbe508353fdd8b3d8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34919 } } }
I20260926 05:51:57.141147 1365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36443d31b41e498fbe508353fdd8b3d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36443d31b41e498fbe508353fdd8b3d8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34919 } } }
I20260926 05:51:57.141345 1365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:57.141546 1366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:57.141700 1373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:57.142365 1373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:57.142759 1362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:57.131201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.24.255.62:34919)
0926 05:51:57.131380 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:57.131388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:57.131412 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:51:57.131949 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:57.131956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:57.132004 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:57.132131 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:57.132892 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.133767 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.134578 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.135261 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.135457 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:57.138328 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:57.138333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:57.138334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:57.138348 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:57.140658 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:57.141096 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:57.141101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:57.141101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:57.141221 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:57.141481 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:57.141482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:57.141482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:57.141822 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:57.141971 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:57.142422 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 05:51:57.142565 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260926 05:51:57.144158 1373 catalog_manager.cc:1347] Generated new cluster ID: ce4a4abf1d9a4fa6b749b0317825fb44
I20260926 05:51:57.144210 1373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:57.157032 1373 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:57.157621 1373 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:57.172266 1373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8: Generated new TSK 0
I20260926 05:51:57.172468 1373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:51:57.205911 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34919
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:57.290937 1385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:57.291148 1385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:57.291173 1385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:57.292364 1385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:57.292409 1385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:57.292435 1385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:57.293960 1385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:57.294227 1385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:57.294428 1385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:57.299665 1391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:57.299659 1390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:57.299775 1385 server_base.cc:1034] running on GCE node
W20260926 05:51:57.299659 1393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:57.300065 1385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:57.300299 1385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:57.301457 1385 hybrid_clock.cc:648] HybridClock initialized: now 1790401917301442 us; error 33 us; skew 500 ppm
Sep 26 05:51:57 dist-test-slave-7fr8 krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:57.304080 1385 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:51:57.304589 1385 webserver.cc:460] Webserver started at http://127.24.255.1:36661/ using document root <none> and password file <none>
I20260926 05:51:57.304816 1385 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:57.304872 1385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:57.305002 1385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:57.306041 1385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "8ceadaf699784056b7a187dcfb1b8d88"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "dde0c7a0ea617bd8bc8cb9ed88c68d23"
server_key_iv: "ad1392e2a12337808cd8524d58b39c9c"
server_key_version: "encryptionkey@0"
I20260926 05:51:57.306488 1385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "8ceadaf699784056b7a187dcfb1b8d88"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "dde0c7a0ea617bd8bc8cb9ed88c68d23"
server_key_iv: "ad1392e2a12337808cd8524d58b39c9c"
server_key_version: "encryptionkey@0"
I20260926 05:51:57.307765 1385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:57.308584 1400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:57.308756 1385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:57.308837 1385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "8ceadaf699784056b7a187dcfb1b8d88"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "dde0c7a0ea617bd8bc8cb9ed88c68d23"
server_key_iv: "ad1392e2a12337808cd8524d58b39c9c"
server_key_version: "encryptionkey@0"
I20260926 05:51:57.308900 1385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:57.318267 1385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:57.320447 1385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:57.320603 1385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:57.320827 1385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:57.321156 1385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:57.321189 1385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:57.321223 1385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:57.321247 1385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:57.326828 1385 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:41757
I20260926 05:51:57.326874 1513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:41757 every 8 connection(s)
I20260926 05:51:57.327188 1385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:51:57.330814 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 1385
I20260926 05:51:57.330935 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:51:57.331120 25596 external_mini_cluster.cc:1428] Setting key f7caed8ac04b51f296a693c7a2eca709
Sep 26 05:51:57 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:57.337997 1362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:57.327899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:51605 (local address 127.24.255.62:34919)
0926 05:51:57.327963 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:57.327965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:57.328158 (+ 193us) server_negotiation.cc:407] Connection header received
0926 05:51:57.328719 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:57.328720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:57.328734 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:57.328781 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:57.329735 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.330481 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.331641 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.331848 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.331905 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:57.334636 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:57.334638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:57.334638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:57.334645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:57.336029 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:57.336587 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:57.336589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:57.336589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:57.336651 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:57.337075 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:57.337077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:57.337077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:57.337250 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:57.337383 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:57.337828 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 05:51:57.337926 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:51:57.338349 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:57.327881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:51605)
0926 05:51:57.328042 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:57.328079 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:57.328615 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:57.328859 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:57.328862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:57.329414 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:57.329635 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:57.329640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.330583 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.330584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:57.331564 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:57.331567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.331923 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.331925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:57.332082 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:57.332632 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:57.332643 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:57.334516 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:57.336132 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:57.336142 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:57.336145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:57.336502 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:57.336717 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:57.336718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:57.336718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:57.336873 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:57.337389 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:57.337450 (+ 61us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:57.337644 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 05:51:57.337780 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 05:51:57.337979 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260926 05:51:57.338974 1514 heartbeater.cc:347] Connected to a master server at 127.24.255.62:34919
I20260926 05:51:57.339072 1514 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:57.339254 1514 heartbeater.cc:511] Master 127.24.255.62:34919 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
I20260926 05:51:57.339818 1323 ts_manager.cc:194] Registered new tserver with Master: 8ceadaf699784056b7a187dcfb1b8d88 (127.24.255.1:41757)
I20260926 05:51:57.340448 1323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:51605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:51:57.361636 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34919
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:57.444154 1519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:57.444371 1519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:57.444403 1519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:57.445626 1519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:57.445674 1519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:57.445710 1519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:57.447292 1519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:57.447557 1519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:57.447781 1519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:57.453044 1524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:57.453011 1525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:57.453029 1527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:57.453328 1519 server_base.cc:1034] running on GCE node
I20260926 05:51:57.453593 1519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:57.453855 1519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:57.455034 1519 hybrid_clock.cc:648] HybridClock initialized: now 1790401917455009 us; error 44 us; skew 500 ppm
Sep 26 05:51:57 dist-test-slave-7fr8 krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:57.457706 1519 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:51:57.458202 1519 webserver.cc:460] Webserver started at http://127.24.255.2:36087/ using document root <none> and password file <none>
I20260926 05:51:57.458421 1519 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:57.458475 1519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:57.458608 1519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:57.459786 1519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "2fad55fb11c94c31b0bdfe820d91bf97"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "3e8bcdfe807aea006fd43b8d8823d474"
server_key_iv: "e83bb46c7ed192c254a725c1f02c9d0b"
server_key_version: "encryptionkey@0"
I20260926 05:51:57.460147 1519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "2fad55fb11c94c31b0bdfe820d91bf97"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "3e8bcdfe807aea006fd43b8d8823d474"
server_key_iv: "e83bb46c7ed192c254a725c1f02c9d0b"
server_key_version: "encryptionkey@0"
I20260926 05:51:57.461429 1519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:51:57.462159 1534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:57.462308 1519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:51:57.462387 1519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "2fad55fb11c94c31b0bdfe820d91bf97"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "3e8bcdfe807aea006fd43b8d8823d474"
server_key_iv: "e83bb46c7ed192c254a725c1f02c9d0b"
server_key_version: "encryptionkey@0"
I20260926 05:51:57.462455 1519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:57.486986 1519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:57.489390 1519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:57.489554 1519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:57.489792 1519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:57.490175 1519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:57.490213 1519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:57.490243 1519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:57.490260 1519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:57.495980 1519 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:43177
I20260926 05:51:57.496065 1647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:43177 every 8 connection(s)
I20260926 05:51:57.496353 1519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:51:57.496934 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 1519
I20260926 05:51:57.497042 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:51:57.497251 25596 external_mini_cluster.cc:1428] Setting key 14a1e7d4aa50c02a45fe11a7a209fe5e
Sep 26 05:51:57 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
I20260926 05:51:57.506547 1362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:57.497114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:60209 (local address 127.24.255.62:34919)
0926 05:51:57.497175 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:57.497177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:57.497425 (+ 248us) server_negotiation.cc:407] Connection header received
0926 05:51:57.497996 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:57.497996 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:57.498008 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:57.498063 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:57.499056 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.499546 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.500865 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.501061 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.501123 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:57.503797 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:57.503799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:57.503799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:57.503808 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:57.505127 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:57.505560 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:57.505562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:57.505562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:57.505614 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:57.505842 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:57.505843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:57.505843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:57.505965 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:57.506020 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:57.506354 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 05:51:57.506450 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:51:57.506764 1650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:57.497096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:60209)
0926 05:51:57.497270 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:57.497315 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:57.497894 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:57.498154 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:57.498159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:57.498726 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:57.498950 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:57.498955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.499630 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.499631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:57.500737 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:57.500741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.501149 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.501150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:57.501300 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:57.501852 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:57.501862 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:57.503690 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:57.505214 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:57.505223 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:57.505226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:57.505457 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:57.505670 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:57.505671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:57.505671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:57.505784 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:57.506025 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:57.506030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:57.506227 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 05:51:57.506323 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 05:51:57.506534 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
I20260926 05:51:57.507262 1648 heartbeater.cc:347] Connected to a master server at 127.24.255.62:34919
I20260926 05:51:57.507362 1648 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:57.507539 1648 heartbeater.cc:511] Master 127.24.255.62:34919 requested a full tablet report, sending...
I20260926 05:51:57.507927 1323 ts_manager.cc:194] Registered new tserver with Master: 2fad55fb11c94c31b0bdfe820d91bf97 (127.24.255.2:43177)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
I20260926 05:51:57.508342 1323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:60209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:51:57.524608 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34919
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:57.604326 1653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:57.604552 1653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:57.604578 1653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:57.605767 1653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:57.605809 1653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:57.605836 1653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:57.607309 1653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:57.607523 1653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:57.607707 1653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:57.612879 1661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:57.612880 1658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:57.612898 1659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:57.613361 1653 server_base.cc:1034] running on GCE node
I20260926 05:51:57.613570 1653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:57.613804 1653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:57.614975 1653 hybrid_clock.cc:648] HybridClock initialized: now 1790401917614948 us; error 48 us; skew 500 ppm
Sep 26 05:51:57 dist-test-slave-7fr8 krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:57.617564 1653 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:51:57.618029 1653 webserver.cc:460] Webserver started at http://127.24.255.3:35759/ using document root <none> and password file <none>
I20260926 05:51:57.618311 1653 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:57.618362 1653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:57.618480 1653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:57.619517 1653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "034496f495d94504946c5dce098421f5"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "0eba2aa093e38c0b588b50cafe3b9989"
server_key_iv: "ecd86092e7248614d4f4169b9fcc204f"
server_key_version: "encryptionkey@0"
I20260926 05:51:57.619812 1653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "034496f495d94504946c5dce098421f5"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "0eba2aa093e38c0b588b50cafe3b9989"
server_key_iv: "ecd86092e7248614d4f4169b9fcc204f"
server_key_version: "encryptionkey@0"
I20260926 05:51:57.621143 1653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:57.621948 1668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:57.622198 1653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:57.622283 1653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "034496f495d94504946c5dce098421f5"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "0eba2aa093e38c0b588b50cafe3b9989"
server_key_iv: "ecd86092e7248614d4f4169b9fcc204f"
server_key_version: "encryptionkey@0"
I20260926 05:51:57.622356 1653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:57.640386 1653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:57.643280 1653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:57.643414 1653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:57.643709 1653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:57.644026 1653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:57.644060 1653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:57.644083 1653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:57.644094 1653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:57.650353 1653 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:41367
I20260926 05:51:57.650431 1781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:41367 every 8 connection(s)
I20260926 05:51:57.650722 1653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:51:57 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:51:57.659859 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 1653
I20260926 05:51:57.659981 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:51:57.660154 25596 external_mini_cluster.cc:1428] Setting key 2490008ab9c9a62172a17ae0d411b3a3
I20260926 05:51:57.661178 1362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:57.651570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:57065 (local address 127.24.255.62:34919)
0926 05:51:57.651627 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:57.651629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:57.651893 (+ 264us) server_negotiation.cc:407] Connection header received
0926 05:51:57.652389 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:57.652390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:57.652403 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:57.652461 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:57.653440 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.653946 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.655005 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.655201 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.655266 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:57.658082 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:57.658084 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:57.658084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:57.658123 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:57.659554 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:57.660027 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:57.660028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:57.660028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:57.660087 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:57.660403 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:57.660404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:57.660404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:57.660551 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:57.660618 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:57.660995 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 05:51:57.661083 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 05:51:57.661437 1784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:57.651493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:57065)
0926 05:51:57.651667 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:57.651742 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:57.652303 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:57.652536 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:57.652540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:57.653120 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:57.653328 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:57.653333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.654035 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.654036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:57.654903 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:57.654906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:57.655285 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:57.655286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:57.655407 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:57.656187 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:57.656200 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:57.657972 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:57.659658 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:57.659668 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:57.659671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:57.659952 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:57.660145 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:57.660146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:57.660146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:57.660267 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:57.660629 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:57.660637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:57.660842 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 05:51:57.660978 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 05:51:57.661177 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260926 05:51:57.661968 1782 heartbeater.cc:347] Connected to a master server at 127.24.255.62:34919
I20260926 05:51:57.662071 1782 heartbeater.cc:464] Registering TS with master...
I20260926 05:51:57.662369 1782 heartbeater.cc:511] Master 127.24.255.62:34919 requested a full tablet report, sending...
I20260926 05:51:57.662798 1323 ts_manager.cc:194] Registered new tserver with Master: 034496f495d94504946c5dce098421f5 (127.24.255.3:41367)
I20260926 05:51:57.663187 1323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:57065
I20260926 05:51:57.672580 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab.
Entry for principal oryx/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab.
WARNING: no policy specified for oryx/127.24.255.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab.
Entry for principal oryx/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab.
WARNING: no policy specified for oryx/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab.
Entry for principal oryx/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab.
I20260926 05:51:57.749022 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 1385
I20260926 05:51:57.753315 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 1519
I20260926 05:51:57.757341 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 1653
I20260926 05:51:57.761431 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 1291
I20260926 05:51:57.766083 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:34919
--webserver_interface=127.24.255.62
--webserver_port=40733
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:34919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:57.849983 1791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:57.850255 1791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:57.850293 1791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:57.851559 1791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:57.851611 1791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:57.851631 1791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:57.851647 1791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:57.851662 1791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:57.853250 1791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:34919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:34919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=40733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:57.853528 1791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:57.853750 1791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:57.858655 1799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:57.858678 1797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:57.858655 1796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:57.859222 1791 server_base.cc:1034] running on GCE node
I20260926 05:51:57.859481 1791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:57.859773 1791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:57.860934 1791 hybrid_clock.cc:648] HybridClock initialized: now 1790401917860921 us; error 27 us; skew 500 ppm
Sep 26 05:51:57 dist-test-slave-7fr8 krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401917, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:57.863559 1791 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:51:57.864013 1791 webserver.cc:460] Webserver started at http://127.24.255.62:40733/ using document root <none> and password file <none>
I20260926 05:51:57.864210 1791 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:57.864266 1791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:57.865633 1791 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:57.866351 1806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:57.866500 1791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:57.866595 1791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "36443d31b41e498fbe508353fdd8b3d8"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "296217833ed8e172cc054abb1674b66c"
server_key_iv: "aa0b5256c31a2d7d6c37a699d02e25fb"
server_key_version: "encryptionkey@0"
I20260926 05:51:57.866991 1791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:57.910935 1791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:57.913120 1791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:57.913244 1791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:57.916960 1791 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:34919
I20260926 05:51:57.916996 1858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:34919 every 8 connection(s)
I20260926 05:51:57.917348 1791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:57.918853 1859 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:51:57.919519 1859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8: Bootstrap starting.
I20260926 05:51:57.921746 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 1791
I20260926 05:51:57.922021 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:41757
--local_ip_for_outbound_sockets=127.24.255.1
--tserver_master_addrs=127.24.255.62:34919
--webserver_port=36661
--webserver_interface=127.24.255.1
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.255.1
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:51:57.928063 1859 log.cc:824] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:57.930349 1859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:51:57.930631 1859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8: Bootstrap complete.
I20260926 05:51:57.932736 1859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36443d31b41e498fbe508353fdd8b3d8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34919 } }
I20260926 05:51:57.933171 1859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36443d31b41e498fbe508353fdd8b3d8, State: Initialized, Role: FOLLOWER
I20260926 05:51:57.933491 1859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36443d31b41e498fbe508353fdd8b3d8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34919 } }
I20260926 05:51:57.933619 1859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:57.933686 1859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:57.933740 1859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:51:57.935288 1859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36443d31b41e498fbe508353fdd8b3d8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34919 } }
I20260926 05:51:57.935424 1859 leader_election.cc:302] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36443d31b41e498fbe508353fdd8b3d8; no voters:
I20260926 05:51:57.935599 1859 leader_election.cc:288] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 05:51:57.935652 1863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:51:57.935796 1863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [term 2 LEADER]: Becoming Leader. State: Replica: 36443d31b41e498fbe508353fdd8b3d8, State: Running, Role: LEADER
I20260926 05:51:57.935936 1859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:57.935954 1863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36443d31b41e498fbe508353fdd8b3d8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34919 } }
I20260926 05:51:57.936256 1864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "36443d31b41e498fbe508353fdd8b3d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36443d31b41e498fbe508353fdd8b3d8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34919 } } }
I20260926 05:51:57.936347 1864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:57.936553 1865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36443d31b41e498fbe508353fdd8b3d8. Latest consensus state: current_term: 2 leader_uuid: "36443d31b41e498fbe508353fdd8b3d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36443d31b41e498fbe508353fdd8b3d8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34919 } } }
I20260926 05:51:57.936650 1865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:57.936779 1869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:57.937276 1869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:57.937417 1869 catalog_manager.cc:1259] Loaded cluster ID: ce4a4abf1d9a4fa6b749b0317825fb44
I20260926 05:51:57.937476 1869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:57.938963 1869 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:57.939145 1869 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 36443d31b41e498fbe508353fdd8b3d8: Loaded TSK: 0
I20260926 05:51:57.939344 1869 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 05:51:58.008488 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:58.008728 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:58.008764 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:58.010057 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:58.010133 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:58.010178 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:51:58.012853 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.1.keytab
--principal=oryx/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:41757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=36661
--tserver_master_addrs=127.24.255.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:58.013178 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:58.013432 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:58.019068 1888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:58.019057 1885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:58.019170 1861 server_base.cc:1034] running on GCE node
W20260926 05:51:58.019057 1886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:58.019595 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:58.019913 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:58.021104 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790401918021085 us; error 33 us; skew 500 ppm
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401918, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:58.024240 1861 init.cc:377] Logged in from keytab as oryx/127.24.255.1@KRBTEST.COM (short username oryx)
I20260926 05:51:58.024868 1861 webserver.cc:460] Webserver started at http://127.24.255.1:36661/ using document root <none> and password file <none>
I20260926 05:51:58.025161 1861 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:58.025246 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:58.027168 1861 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:58.027899 1895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:58.028107 1861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:58.028209 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "8ceadaf699784056b7a187dcfb1b8d88"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "dde0c7a0ea617bd8bc8cb9ed88c68d23"
server_key_iv: "ad1392e2a12337808cd8524d58b39c9c"
server_key_version: "encryptionkey@0"
I20260926 05:51:58.028643 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:58.039279 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:58.042004 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:58.042207 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:58.042479 1861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:58.042847 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:58.042896 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:58.042945 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:58.042968 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:58.051635 1861 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:41757
I20260926 05:51:58.051669 2008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:41757 every 8 connection(s)
I20260926 05:51:58.052114 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:51:58.057449 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 1861
I20260926 05:51:58.057807 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:43177
--local_ip_for_outbound_sockets=127.24.255.2
--tserver_master_addrs=127.24.255.62:34919
--webserver_port=36087
--webserver_interface=127.24.255.2
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.255.2
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:51:58.062657 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:58.052980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:32805)
0926 05:51:58.053190 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:58.053238 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:58.053886 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:58.054399 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:58.054404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:58.055193 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:58.055493 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.055499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.056710 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.056712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.057992 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.057996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.058826 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.058828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.058980 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.059511 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:58.059523 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:58.062425 (+ 2902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
W20260926 05:51:58.062825 2012 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:58.053094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:32805 (local address 127.24.255.62:34919)
0926 05:51:58.053267 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:58.053275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:58.053345 (+ 70us) server_negotiation.cc:407] Connection header received
0926 05:51:58.054152 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:58.054160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:58.054217 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:58.054325 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:58.055631 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.056592 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.058115 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.058715 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.058835 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.062623 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:32805: BlockingRecv error: recv got EOF from 127.24.255.1:32805 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
W20260926 05:51:58.063089 2009 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:34919 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.255.62:34919: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:51:58.069329 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:58.063360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:44619)
0926 05:51:58.063412 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:58.063422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:58.063470 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:58.063617 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:58.063619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:58.063774 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:58.063840 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.063842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.064960 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.064961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.065775 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.065778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.066271 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.066273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.066352 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.066834 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:58.066839 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:58.069249 (+ 2410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 05:51:58.069432 2012 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:58.063389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:44619 (local address 127.24.255.62:34919)
0926 05:51:58.063426 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:58.063428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:58.063460 (+ 32us) server_negotiation.cc:407] Connection header received
0926 05:51:58.063525 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:58.063526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:58.063538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:58.063601 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:58.063911 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.064862 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.065866 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.066193 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.066263 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.069366 (+ 3103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:44619: BlockingRecv error: recv got EOF from 127.24.255.1:44619 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:51:58.075117 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:58.069582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:35207)
0926 05:51:58.069821 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:58.069832 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:58.069878 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:58.070062 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:58.070063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:58.070229 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:58.070269 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.070271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.071164 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.071165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.071996 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.071998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.072461 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.072462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.072523 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.072999 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:58.073004 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:58.075034 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":89,"threads_started":1}
W20260926 05:51:58.075255 2009 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:51:58.075374 2012 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:58.069772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:35207 (local address 127.24.255.62:34919)
0926 05:51:58.069832 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:58.069835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:58.069865 (+ 30us) server_negotiation.cc:407] Connection header received
0926 05:51:58.069969 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:58.069970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:58.069984 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:58.070048 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:58.070346 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.071095 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.072074 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.072388 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.072460 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.075138 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:35207: BlockingRecv error: recv got EOF from 127.24.255.1:35207 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 05:51:58.146924 2013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:58.147154 2013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:58.147181 2013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:58.148654 2013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:58.148718 2013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:58.148748 2013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:51:58.150503 2013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.2.keytab
--principal=oryx/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:43177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=36087
--tserver_master_addrs=127.24.255.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:58.150753 2013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:58.150988 2013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:58.156246 2019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:58.156246 2022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:58.156416 2013 server_base.cc:1034] running on GCE node
W20260926 05:51:58.156286 2020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:58.156716 2013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:58.157089 2013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:58.158252 2013 hybrid_clock.cc:648] HybridClock initialized: now 1790401918158232 us; error 37 us; skew 500 ppm
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401918, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:58.160897 2013 init.cc:377] Logged in from keytab as oryx/127.24.255.2@KRBTEST.COM (short username oryx)
I20260926 05:51:58.161468 2013 webserver.cc:460] Webserver started at http://127.24.255.2:36087/ using document root <none> and password file <none>
I20260926 05:51:58.161723 2013 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:58.161789 2013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:58.163301 2013 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:58.163983 2029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:58.164192 2013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:58.164341 2013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "2fad55fb11c94c31b0bdfe820d91bf97"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "3e8bcdfe807aea006fd43b8d8823d474"
server_key_iv: "e83bb46c7ed192c254a725c1f02c9d0b"
server_key_version: "encryptionkey@0"
I20260926 05:51:58.164701 2013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:58.180624 2013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:58.183081 2013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:58.183234 2013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:58.183455 2013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:58.183753 2013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:58.183787 2013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:58.183817 2013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:58.183835 2013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:58.189646 2013 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:43177
I20260926 05:51:58.189700 2142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:43177 every 8 connection(s)
I20260926 05:51:58.190057 2013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:51:58.194046 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 2013
I20260926 05:51:58.194423 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:41367
--local_ip_for_outbound_sockets=127.24.255.3
--tserver_master_addrs=127.24.255.62:34919
--webserver_port=35759
--webserver_interface=127.24.255.3
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.255.3
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 05:51:58.201015 2012 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:58.191214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:49303 (local address 127.24.255.62:34919)
0926 05:51:58.191338 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:58.191341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:58.191467 (+ 126us) server_negotiation.cc:407] Connection header received
0926 05:51:58.192100 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:58.192102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:58.192118 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:58.192178 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:58.193976 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.194646 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.196518 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.196855 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.196979 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.200955 (+ 3976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:49303: BlockingRecv error: recv got EOF from 127.24.255.2:49303 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 05:51:58.201054 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:58.191089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:49303)
0926 05:51:58.191292 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:58.191346 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:58.191976 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:58.192326 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:58.192330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:58.193447 (+ 1117us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:58.193780 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.193792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.194787 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.194790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.196343 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.196358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.197015 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.197017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.197173 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.197780 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:58.197791 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:58.200815 (+ 3024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
W20260926 05:51:58.201284 2143 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:34919 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.255.62:34919: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:51:58.207531 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:58.201591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:60091)
0926 05:51:58.201678 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:58.201690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:58.201744 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:58.201913 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:58.201914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:58.202075 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:58.202177 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.202179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.203127 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.203128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.203676 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.203678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.204137 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.204138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.204196 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.204680 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:58.204684 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:58.207448 (+ 2764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 05:51:58.207616 2012 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:58.201679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:60091 (local address 127.24.255.62:34919)
0926 05:51:58.201749 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:58.201751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:58.201764 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:51:58.201800 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:58.201801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:58.201812 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:58.201877 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:58.202252 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.203008 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.203759 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.204053 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.204128 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.207543 (+ 3415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:60091: BlockingRecv error: recv got EOF from 127.24.255.2:60091 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:51:58.214164 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:58.207880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:52449)
0926 05:51:58.207924 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:58.207933 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:58.207969 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:58.208118 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:58.208119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:58.208262 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:58.208306 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.208308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.208994 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.208994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.210130 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.210134 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.210886 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.210887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.211026 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.211743 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:58.211748 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:58.214041 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 05:51:58.214253 2012 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:58.207939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:52449 (local address 127.24.255.62:34919)
0926 05:51:58.207995 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:58.207997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:58.208009 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:51:58.208043 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:58.208044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:58.208052 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:58.208099 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:58.208377 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.208921 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.210470 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.210783 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.210854 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.214198 (+ 3344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:52449: BlockingRecv error: recv got EOF from 127.24.255.2:52449 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 05:51:58.214504 2143 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:51:58.284837 2146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:58.285073 2146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:58.285101 2146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:58.286554 2146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:58.286620 2146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:58.286651 2146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:51:58.288290 2146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/krb5kdc/oryx_127.24.255.3.keytab
--principal=oryx/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:41367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=35759
--tserver_master_addrs=127.24.255.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:58.288512 2146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:58.288707 2146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:58.293928 2154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:58.293922 2152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:58.293941 2151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:58.293965 2146 server_base.cc:1034] running on GCE node
I20260926 05:51:58.294441 2146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:58.294709 2146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:58.295876 2146 hybrid_clock.cc:648] HybridClock initialized: now 1790401918295820 us; error 69 us; skew 500 ppm
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401918, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:58.298570 2146 init.cc:377] Logged in from keytab as oryx/127.24.255.3@KRBTEST.COM (short username oryx)
I20260926 05:51:58.299149 2146 webserver.cc:460] Webserver started at http://127.24.255.3:35759/ using document root <none> and password file <none>
I20260926 05:51:58.299445 2146 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:58.299517 2146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:58.300922 2146 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:58.301585 2161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:58.301748 2146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:51:58.301831 2146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "034496f495d94504946c5dce098421f5"
format_stamp: "Formatted at 2026-09-26 05:51:57 on dist-test-slave-7fr8"
server_key: "0eba2aa093e38c0b588b50cafe3b9989"
server_key_iv: "ecd86092e7248614d4f4169b9fcc204f"
server_key_version: "encryptionkey@0"
I20260926 05:51:58.302208 2146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:58.314460 2146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:58.316576 2146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:58.316718 2146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:58.316941 2146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:51:58.317350 2146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:51:58.317391 2146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:58.317422 2146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:51:58.317448 2146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:58.323277 2146 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:41367
I20260926 05:51:58.323321 2274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:41367 every 8 connection(s)
I20260926 05:51:58.323663 2146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:51:58.331372 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 2146
W20260926 05:51:58.331521 2012 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:58.324361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:55017 (local address 127.24.255.62:34919)
0926 05:51:58.324437 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:58.324439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:58.324638 (+ 199us) server_negotiation.cc:407] Connection header received
0926 05:51:58.325162 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:58.325163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:58.325174 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:58.325230 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:58.326326 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.326812 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.327860 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.328074 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.328146 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.331455 (+ 3309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:55017: BlockingRecv error: recv got EOF from 127.24.255.3:55017 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:51:58.331560 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:58.324336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:55017)
0926 05:51:58.324520 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:58.324559 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:58.325059 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:58.325331 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:58.325335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:58.325926 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:58.326223 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.326229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.326890 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.326891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.327760 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.327763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.328262 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.328264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.328516 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.329052 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:58.329061 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:58.331339 (+ 2278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
W20260926 05:51:58.331807 2275 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:34919 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.255.62:34919: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:51:58.337185 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:58.332043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:37065)
0926 05:51:58.332148 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:58.332158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:58.332191 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:58.332334 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:58.332335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:58.332477 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:58.332541 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.332544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.333243 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.333244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.333903 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.333905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.334425 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.334426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.334488 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.334898 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:58.334903 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:58.337110 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260926 05:51:58.337282 2279 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:58.332073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:37065 (local address 127.24.255.62:34919)
0926 05:51:58.332231 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:58.332233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:58.332240 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:58.332256 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:58.332256 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:58.332265 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:58.332323 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:58.332636 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.333173 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.333993 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.334331 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.334411 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.337212 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:37065: BlockingRecv error: recv got EOF from 127.24.255.3:37065 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":45,"threads_started":1}
I20260926 05:51:58.339833 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:58.331887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56118 (local address 127.24.255.62:34919)
0926 05:51:58.331927 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:58.331929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:58.332047 (+ 118us) server_negotiation.cc:407] Connection header received
0926 05:51:58.332216 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:58.332218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:58.332231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:58.332295 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:58.332937 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.333564 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.334600 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.334830 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.334897 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.336235 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:58.336240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:58.336241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:58.336257 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:58.338283 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:58.338697 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:58.338700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:58.338700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:58.338785 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:58.338962 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:58.338963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:58.338963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:58.339256 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:58.339395 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:58.339623 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 05:51:58.339738 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:58 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:51:58.342545 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:58.337598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:52391)
0926 05:51:58.337644 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:58.337654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:58.337689 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:58.337899 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:58.337900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:58.338058 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:58.338128 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.338131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.338976 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.338977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.339698 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:58.339701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.340119 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.340120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:58.340179 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.340577 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:58.340582 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:58.342491 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 05:51:58.342633 2279 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:58.337671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:52391 (local address 127.24.255.62:34919)
0926 05:51:58.337713 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:58.337715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:58.337724 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:58.337742 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:58.337743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:58.337751 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:58.337888 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:58.338207 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.338882 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.339779 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:58.340045 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:58.340098 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:58.342578 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:52391: BlockingRecv error: recv got EOF from 127.24.255.3:52391 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 05:51:58.342733 2275 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 05:51:59 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:59 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:51:59.081060 2280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:59.075716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:51019)
0926 05:51:59.075875 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:59.075886 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:59.075938 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:59.076138 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:59.076139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:59.076294 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:59.076342 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:59.076343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:59.077241 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:59.077242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:59.077806 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:59.077808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:59.078339 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:59.078340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:59.078395 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:59.078986 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:59.078997 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:59.080986 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":61,"threads_started":1}
W20260926 05:51:59.081190 2281 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:59.075888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:51019 (local address 127.24.255.62:34919)
0926 05:51:59.076008 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:59.076010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:59.076019 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:59.076045 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:59.076046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:59.076055 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:59.076127 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:59.076484 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:59.077106 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:59.077906 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:59.078221 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:59.078294 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:59.081118 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:51019: BlockingRecv error: recv got EOF from 127.24.255.1:51019 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
Sep 26 05:51:59 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:59 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:51:59.219988 2282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:59.214971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:39659)
0926 05:51:59.215125 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:59.215139 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:59.215214 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:59.215356 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:59.215357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:59.215528 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:59.215572 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:59.215574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:59.216272 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:59.216273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:59.216888 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:59.216891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:59.217295 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:59.217296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:59.217348 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:59.217839 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:59.217844 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:59.219899 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
W20260926 05:51:59.220108 2281 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:59.215099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:39659 (local address 127.24.255.62:34919)
0926 05:51:59.215164 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:59.215166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:59.215195 (+ 29us) server_negotiation.cc:407] Connection header received
0926 05:51:59.215278 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:59.215282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:59.215293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:59.215345 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:59.215645 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:59.216164 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:59.216982 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:59.217177 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:59.217253 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:59.220038 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:39659: BlockingRecv error: recv got EOF from 127.24.255.2:39659 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 05:51:59 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:51:59 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:51:59.347811 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:59.343206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:55421)
0926 05:51:59.343350 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:59.343361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:59.343412 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:59.343583 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:59.343584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:59.343743 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:59.343792 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:59.343794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:59.344456 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:59.344456 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:59.345015 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:59.345017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:59.345350 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:59.345350 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:59.345401 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:59.345916 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:59.345922 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:59.347739 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":60,"threads_started":1}
W20260926 05:51:59.347905 2281 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:59.343333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:55421 (local address 127.24.255.62:34919)
0926 05:51:59.343453 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:59.343454 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:59.343465 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:51:59.343492 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:59.343493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:59.343507 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:59.343570 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:59.343849 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:59.344340 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:59.345082 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:59.345289 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:59.345345 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:59.347842 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:55421: BlockingRecv error: recv got EOF from 127.24.255.3:55421 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 26 05:52:00 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:00 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:00.086135 2284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:00.081544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:33121)
0926 05:52:00.081664 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:00.081677 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:00.081739 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:00.081873 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:00.081874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:00.082030 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:00.082084 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:00.082085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:00.082780 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:00.082781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:00.083343 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:00.083346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:00.083757 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:00.083758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:00.083803 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:00.084246 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:00.084250 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:00.086025 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
W20260926 05:52:00.086216 2285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:00.081631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:33121 (local address 127.24.255.62:34919)
0926 05:52:00.081756 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:00.081758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:00.081767 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:52:00.081788 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:00.081788 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:00.081798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:00.081852 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:00.082177 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:00.082688 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:00.083480 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:00.083683 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:00.083743 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:00.086154 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:33121: BlockingRecv error: recv got EOF from 127.24.255.1:33121 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
Sep 26 05:52:00 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:00 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:00.225787 2286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:00.220584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:49231)
0926 05:52:00.220719 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:00.220745 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:00.220824 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:00.220989 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:00.220989 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:00.221151 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:00.221195 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:00.221197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:00.221857 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:00.221858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:00.222383 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:00.222385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:00.222726 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:00.222727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:00.222780 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:00.223337 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:00.223342 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:00.225690 (+ 2348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
W20260926 05:52:00.225858 2285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:00.220709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:49231 (local address 127.24.255.62:34919)
0926 05:52:00.220803 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:00.220805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:00.220816 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:52:00.220893 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:00.220894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:00.220906 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:00.220976 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:00.221266 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:00.221778 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:00.222461 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:00.222649 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:00.222720 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:00.225804 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:49231: BlockingRecv error: recv got EOF from 127.24.255.2:49231 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 05:52:00 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:00 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:00.353066 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:00.348315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:37179)
0926 05:52:00.348465 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:00.348477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:00.348529 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:00.348678 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:00.348678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:00.348849 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:00.348899 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:00.348901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:00.349668 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:00.349669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:00.350252 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:00.350254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:00.350583 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:00.350584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:00.350636 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:00.351090 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:00.351095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:00.352981 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
W20260926 05:52:00.353147 2285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:00.348454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:37179 (local address 127.24.255.62:34919)
0926 05:52:00.348572 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:00.348574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:00.348582 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:52:00.348604 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:00.348604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:00.348616 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:00.348660 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:00.348987 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:00.349554 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:00.350335 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:00.350523 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:00.350579 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:00.353090 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:37179: BlockingRecv error: recv got EOF from 127.24.255.3:37179 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 26 05:52:01 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:01 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:01.091080 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:01.086716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:51005)
0926 05:52:01.086875 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:01.086885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:01.086935 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:01.087068 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:01.087069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:01.087225 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:01.087274 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:01.087275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:01.087963 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:01.087964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:01.088504 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:01.088506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:01.088838 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:01.088838 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:01.088873 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:01.089279 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:01.089283 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:01.091003 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
W20260926 05:52:01.091248 2289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:01.086813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:51005 (local address 127.24.255.62:34919)
0926 05:52:01.086931 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:01.086933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:01.086943 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:52:01.086972 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:01.086973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:01.086982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:01.087052 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:01.087345 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:01.087893 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:01.088588 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:01.088784 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:01.088833 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:01.091110 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:51005: BlockingRecv error: recv got EOF from 127.24.255.1:51005 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
Sep 26 05:52:01 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:01 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:01.231233 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:01.226321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:34933)
0926 05:52:01.226531 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:01.226542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:01.226605 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:01.226728 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:01.226729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:01.226900 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:01.226949 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:01.226951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:01.227644 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:01.227645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:01.228238 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:01.228241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:01.228625 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:01.228626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:01.228672 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:01.229153 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:01.229158 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:01.231153 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":64,"threads_started":1}
W20260926 05:52:01.231350 2289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:01.226425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:34933 (local address 127.24.255.62:34919)
0926 05:52:01.226531 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:01.226533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:01.226600 (+ 67us) server_negotiation.cc:407] Connection header received
0926 05:52:01.226660 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:01.226660 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:01.226670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:01.226718 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:01.227029 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:01.227545 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:01.228336 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:01.228552 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:01.228616 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:01.231281 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:34933: BlockingRecv error: recv got EOF from 127.24.255.2:34933 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 05:52:01 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:01 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:01.359196 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:01.353857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:52819)
0926 05:52:01.354031 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:01.354045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:01.354138 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:01.354370 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:01.354371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:01.354529 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:01.354582 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:01.354584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:01.355239 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:01.355240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:01.355765 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:01.355767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:01.356118 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:01.356119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:01.356172 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:01.356806 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:01.356811 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:01.359097 (+ 2286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
W20260926 05:52:01.359302 2289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:01.353990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:52819 (local address 127.24.255.62:34919)
0926 05:52:01.354046 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:01.354048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:01.354120 (+ 72us) server_negotiation.cc:407] Connection header received
0926 05:52:01.354226 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:01.354227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:01.354241 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:01.354291 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:01.354657 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:01.355165 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:01.355847 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:01.356057 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:01.356111 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:01.359228 (+ 3117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:52819: BlockingRecv error: recv got EOF from 127.24.255.3:52819 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 26 05:52:02 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:02 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:02.096385 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:02.091549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:40131)
0926 05:52:02.091686 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:02.091697 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:02.091742 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:02.091951 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:02.091952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:02.092114 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:02.092160 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:02.092162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:02.092902 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:02.092904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:02.093498 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:02.093501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:02.093997 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:02.093998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:02.094046 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:02.094536 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:02.094540 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:02.096311 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
W20260926 05:52:02.096580 2293 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:02.091662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:40131 (local address 127.24.255.62:34919)
0926 05:52:02.091774 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:02.091775 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:02.091784 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:52:02.091808 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:02.091808 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:02.091818 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:02.091881 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:02.092262 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:02.092792 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:02.093670 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:02.093887 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:02.093949 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:02.096500 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:40131: BlockingRecv error: recv got EOF from 127.24.255.1:40131 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":52,"threads_started":1}
Sep 26 05:52:02 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:02 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:02.243558 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:02.231761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:33277)
0926 05:52:02.231885 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:02.231894 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:02.231964 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:02.232174 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:02.232175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:02.232406 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:02.232456 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:02.232458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:02.233162 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:02.233163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:02.233826 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:02.233829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:02.234554 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:02.234555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:02.234608 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:02.241564 (+ 6956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:02.241570 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:02.243495 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
W20260926 05:52:02.243733 2293 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:02.231808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:33277 (local address 127.24.255.62:34919)
0926 05:52:02.231864 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:02.231867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:02.232001 (+ 134us) server_negotiation.cc:407] Connection header received
0926 05:52:02.232032 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:02.232033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:02.232045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:02.232163 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:02.232572 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:02.233082 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:02.233974 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:02.234317 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:02.234546 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:02.243665 (+ 9119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:33277: BlockingRecv error: recv got EOF from 127.24.255.2:33277 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 05:52:02 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:02 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:02.365334 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:02.359764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:32787)
0926 05:52:02.359906 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:02.359919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:02.359974 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:02.360165 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:02.360166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:02.360334 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:02.360392 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:02.360394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:02.361183 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:02.361185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:02.361731 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:02.361733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:02.362385 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:02.362386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:02.362441 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:02.362945 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:02.362950 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:02.365234 (+ 2284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
W20260926 05:52:02.365495 2293 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:02.359922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:32787 (local address 127.24.255.62:34919)
0926 05:52:02.359992 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:02.359993 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:02.360003 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:52:02.360032 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:02.360033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:02.360046 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:02.360114 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:02.360492 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:02.361055 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:02.361918 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:02.362243 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:02.362354 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:02.365397 (+ 3043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:32787: BlockingRecv error: recv got EOF from 127.24.255.3:32787 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 05:52:03 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:03 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:03.103188 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:03.097047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:40493)
0926 05:52:03.097201 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:03.097213 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:03.097278 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:03.097565 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:03.097568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:03.097807 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:03.097886 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:03.097888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:03.098983 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:03.098985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:03.099565 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:03.099568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:03.100283 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:03.100285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:03.100359 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:03.101009 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:03.101015 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:03.103119 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
W20260926 05:52:03.103488 2297 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:03.097154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:40493 (local address 127.24.255.62:34919)
0926 05:52:03.097339 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:03.097341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:03.097351 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:52:03.097373 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:03.097374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:03.097386 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:03.097460 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:03.098039 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:03.098740 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:03.099727 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:03.100003 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:03.100122 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:03.103428 (+ 3306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:40493: BlockingRecv error: recv got EOF from 127.24.255.1:40493 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
Sep 26 05:52:03 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:03 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:03.250878 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:03.244421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:47177)
0926 05:52:03.244772 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:03.244790 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:03.244850 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:03.245079 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:03.245081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:03.245370 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:03.245485 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:03.245489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:03.246601 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:03.246603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:03.247424 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:03.247430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:03.247834 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:03.247835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:03.247886 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:03.248542 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:03.248548 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:03.250792 (+ 2244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":105,"threads_started":1}
W20260926 05:52:03.251160 2297 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:03.244682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:47177 (local address 127.24.255.62:34919)
0926 05:52:03.244821 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:03.244823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:03.244837 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:52:03.244901 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:03.244903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:03.244915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:03.244983 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:03.245577 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:03.246393 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:03.247549 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:03.247762 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:03.247825 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:03.250945 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:47177: BlockingRecv error: recv got EOF from 127.24.255.2:47177 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 26 05:52:03 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:03 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:03.372098 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:03.365964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:38887)
0926 05:52:03.366165 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:03.366176 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:03.366247 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:03.366440 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:03.366441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:03.366616 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:03.366666 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:03.366668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:03.367360 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:03.367361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:03.367962 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:03.367964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:03.368499 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:03.368501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:03.368588 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:03.369257 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:03.369264 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:03.372026 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
W20260926 05:52:03.372267 2297 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:03.366133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:38887 (local address 127.24.255.62:34919)
0926 05:52:03.366202 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:03.366204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:03.366233 (+ 29us) server_negotiation.cc:407] Connection header received
0926 05:52:03.366358 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:03.366359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:03.366373 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:03.366429 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:03.366738 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:03.367244 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:03.368066 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:03.368337 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:03.368403 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:03.372171 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:38887: BlockingRecv error: recv got EOF from 127.24.255.3:38887 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 05:52:04 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:04 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:04.109022 2300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:04.103864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:49373)
0926 05:52:04.104037 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:04.104053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:04.104142 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:04.104406 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:04.104408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:04.104568 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:04.104619 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:04.104620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:04.105412 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:04.105413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:04.105952 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:04.105954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:04.106515 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:04.106516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:04.106579 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:04.107070 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:04.107075 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:04.108938 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
W20260926 05:52:04.109242 2301 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:04.104017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:49373 (local address 127.24.255.62:34919)
0926 05:52:04.104188 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:04.104190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:04.104201 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:52:04.104230 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:04.104231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:04.104243 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:04.104323 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:04.104773 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:04.105334 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:04.106025 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:04.106346 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:04.106500 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:04.109046 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:49373: BlockingRecv error: recv got EOF from 127.24.255.1:49373 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
Sep 26 05:52:04 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:04 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:04.256587 2302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:04.251611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:38009)
0926 05:52:04.251730 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:04.251741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:04.251800 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:04.252074 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:04.252075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:04.252233 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:04.252290 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:04.252293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:04.253141 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:04.253143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:04.253688 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:04.253690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:04.254118 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:04.254119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:04.254202 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:04.254655 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:04.254660 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:04.256523 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
W20260926 05:52:04.257045 2301 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:04.251679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:38009 (local address 127.24.255.62:34919)
0926 05:52:04.251734 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:04.251736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:04.251788 (+ 52us) server_negotiation.cc:407] Connection header received
0926 05:52:04.251943 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:04.251944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:04.251957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:04.252026 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:04.252445 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:04.253059 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:04.253794 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:04.254002 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:04.254065 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:04.256928 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:38009: BlockingRecv error: recv got EOF from 127.24.255.2:38009 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 26 05:52:04 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:04 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:04.378196 2303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:04.372722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:37019)
0926 05:52:04.372887 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:04.372898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:04.373002 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:04.373253 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:04.373254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:04.373468 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:04.373519 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:04.373521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:04.374341 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:04.374342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:04.374996 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:04.374999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:04.375394 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:04.375395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:04.375445 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:04.375915 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:04.375921 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:04.378120 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
W20260926 05:52:04.378298 2301 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:04.372853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:37019 (local address 127.24.255.62:34919)
0926 05:52:04.372948 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:04.372950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:04.372962 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:52:04.373138 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:04.373139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:04.373152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:04.373211 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:04.373616 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:04.374239 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:04.375077 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:04.375315 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:04.375374 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:04.378225 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:37019: BlockingRecv error: recv got EOF from 127.24.255.3:37019 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 05:52:05 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:05 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:05.114856 2304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:05.109643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:44403)
0926 05:52:05.109777 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:05.109786 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:05.109838 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:05.110132 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:05.110133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:05.110353 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:05.110403 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:05.110405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:05.111144 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:05.111145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:05.111775 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:05.111778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:05.112272 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:05.112273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:05.112331 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:05.112843 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:05.112848 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:05.114756 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
W20260926 05:52:05.114948 2305 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:05.109768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:44403 (local address 127.24.255.62:34919)
0926 05:52:05.109947 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:05.109949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:05.109959 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:52:05.109985 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:05.109986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:05.109997 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:05.110067 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:05.110495 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:05.111033 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:05.111990 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:05.112206 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:05.112263 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:05.114885 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:44403: BlockingRecv error: recv got EOF from 127.24.255.1:44403 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":73,"threads_started":1}
Sep 26 05:52:05 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:05 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:05.262423 2306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:05.257085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:34219)
0926 05:52:05.257231 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:05.257242 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:05.257290 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:05.257485 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:05.257486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:05.257638 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:05.257684 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:05.257685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:05.258470 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:05.258471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:05.259013 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:05.259015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:05.259425 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:05.259427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:05.259475 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:05.259967 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:05.259972 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:05.262359 (+ 2387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
W20260926 05:52:05.262679 2305 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:05.257191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:34219 (local address 127.24.255.62:34919)
0926 05:52:05.257289 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:05.257291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:05.257329 (+ 38us) server_negotiation.cc:407] Connection header received
0926 05:52:05.257413 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:05.257414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:05.257425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:05.257469 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:05.257780 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:05.258374 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:05.259109 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:05.259307 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:05.259369 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:05.262602 (+ 3233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:34219: BlockingRecv error: recv got EOF from 127.24.255.2:34219 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 05:52:05 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:05 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:05.383296 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:05.378725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:39121)
0926 05:52:05.378921 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:05.378931 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:05.378989 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:05.379146 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:05.379147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:05.379324 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:05.379376 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:05.379377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:05.380031 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:05.380032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:05.380557 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:05.380559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:05.380886 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:05.380887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:05.380938 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:05.381365 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:05.381370 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:05.383234 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
W20260926 05:52:05.383721 2305 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:05.378777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:39121 (local address 127.24.255.62:34919)
0926 05:52:05.378836 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:05.378838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:05.378989 (+ 151us) server_negotiation.cc:407] Connection header received
0926 05:52:05.379066 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:05.379067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:05.379082 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:05.379130 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:05.379458 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:05.379955 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:05.380641 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:05.380822 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:05.380877 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:05.383663 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:39121: BlockingRecv error: recv got EOF from 127.24.255.3:39121 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 26 05:52:06 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:06 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:06.120512 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:06.115532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:33797)
0926 05:52:06.115676 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:06.115686 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:06.115736 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:06.116039 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:06.116040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:06.116219 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:06.116265 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:06.116267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:06.117028 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:06.117029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:06.117601 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:06.117603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:06.118033 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:06.118034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:06.118085 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:06.118578 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:06.118583 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:06.120449 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
W20260926 05:52:06.120675 2309 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:06.115687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:33797 (local address 127.24.255.62:34919)
0926 05:52:06.115885 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:06.115887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:06.115893 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:52:06.115917 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:06.115918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:06.115928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:06.115981 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:06.116377 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:06.116939 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:06.117703 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:06.117921 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:06.117985 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:06.120602 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:33797: BlockingRecv error: recv got EOF from 127.24.255.1:33797 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
Sep 26 05:52:06 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:06 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:06.268687 2310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:06.263062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:58867)
0926 05:52:06.263185 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:06.263196 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:06.263245 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:06.263502 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:06.263504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:06.263728 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:06.263799 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:06.263801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:06.264934 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:06.264936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:06.265491 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:06.265493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:06.265948 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:06.265949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:06.265997 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:06.266612 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:06.266617 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:06.268620 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
W20260926 05:52:06.268851 2309 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:06.263188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:58867 (local address 127.24.255.62:34919)
0926 05:52:06.263244 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:06.263245 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:06.263297 (+ 52us) server_negotiation.cc:407] Connection header received
0926 05:52:06.263333 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:06.263334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:06.263345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:06.263391 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:06.263959 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:06.264796 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:06.265667 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:06.265868 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:06.265933 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:06.268747 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:58867: BlockingRecv error: recv got EOF from 127.24.255.2:58867 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 05:52:06 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:06 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:06.389019 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:06.383825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:34299)
0926 05:52:06.383968 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:06.383979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:06.384049 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:06.384233 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:06.384234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:06.384407 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:06.384456 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:06.384458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:06.385141 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:06.385142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:06.385725 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:06.385727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:06.386270 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:06.386272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:06.386332 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:06.387113 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:06.387118 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:06.388938 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
W20260926 05:52:06.389089 2309 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:06.383936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:34299 (local address 127.24.255.62:34919)
0926 05:52:06.384032 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:06.384034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:06.384047 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:52:06.384111 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:06.384112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:06.384123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:06.384197 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:06.384539 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:06.385049 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:06.385861 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:06.386056 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:06.386236 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:06.389034 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:34299: BlockingRecv error: recv got EOF from 127.24.255.3:34299 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 05:52:07 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:07 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:07.127488 2312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:07.121145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:41911)
0926 05:52:07.121300 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:07.121310 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:07.121431 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:07.121627 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:07.121628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:07.121789 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:07.121837 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:07.121839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:07.122620 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:07.122621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:07.123254 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:07.123257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:07.123943 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:07.123945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:07.124015 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:07.124650 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:07.124655 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:07.127383 (+ 2728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
W20260926 05:52:07.127591 2313 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:07.121261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:41911 (local address 127.24.255.62:34919)
0926 05:52:07.121418 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:07.121420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:07.121432 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:52:07.121493 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:07.121494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:07.121505 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:07.121577 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:07.121944 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:07.122527 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:07.123437 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:07.123796 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:07.123888 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:07.127520 (+ 3632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:41911: BlockingRecv error: recv got EOF from 127.24.255.1:41911 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":62,"threads_started":1}
Sep 26 05:52:07 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:07 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:07.278427 2314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:07.269060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:53389)
0926 05:52:07.269216 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:07.269227 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:07.269282 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:07.269462 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:07.269463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:07.269621 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:07.269669 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:07.269671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:07.270519 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:07.270520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:07.271055 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:07.271057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:07.271454 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:07.271455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:07.271505 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:07.276204 (+ 4699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:07.276212 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:07.278365 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
W20260926 05:52:07.278599 2313 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:07.269210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:53389 (local address 127.24.255.62:34919)
0926 05:52:07.269264 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:07.269266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:07.269274 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:52:07.269344 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:07.269344 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:07.269356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:07.269407 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:07.269840 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:07.270411 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:07.271148 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:07.271347 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:07.271411 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:07.278538 (+ 7127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:53389: BlockingRecv error: recv got EOF from 127.24.255.2:53389 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 05:52:07 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:07 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:07.395300 2315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:07.389895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:45293)
0926 05:52:07.390242 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:07.390255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:07.390328 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:07.390494 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:07.390494 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:07.390708 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:07.390768 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:07.390771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:07.391565 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:07.391566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:07.392112 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:07.392113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:07.392568 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:07.392569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:07.392618 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:07.393349 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:07.393355 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:07.395225 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":105,"threads_started":1}
W20260926 05:52:07.395432 2313 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:07.389970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:45293 (local address 127.24.255.62:34919)
0926 05:52:07.390148 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:07.390150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:07.390323 (+ 173us) server_negotiation.cc:407] Connection header received
0926 05:52:07.390405 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:07.390406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:07.390422 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:07.390479 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:07.390875 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:07.391464 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:07.392225 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:07.392473 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:07.392538 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:07.395364 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:45293: BlockingRecv error: recv got EOF from 127.24.255.3:45293 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Sep 26 05:52:08 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:08 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:08.133044 2316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:08.128114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:39131)
0926 05:52:08.128275 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:08.128286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:08.128340 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:08.128470 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:08.128471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:08.128647 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:08.128694 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:08.128696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:08.129390 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:08.129391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:08.129909 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:08.129910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:08.130469 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:08.130470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:08.130520 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:08.131028 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:08.131033 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:08.132980 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":68,"threads_started":1}
W20260926 05:52:08.133226 2317 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:08.128211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:39131 (local address 127.24.255.62:34919)
0926 05:52:08.128326 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:08.128328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:08.128336 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:52:08.128385 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:08.128385 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:08.128396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:08.128451 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:08.128784 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:08.129311 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:08.130064 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:08.130353 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:08.130449 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:08.133153 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:39131: BlockingRecv error: recv got EOF from 127.24.255.1:39131 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":47,"threads_started":1}
Sep 26 05:52:08 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:08 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:08.285181 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:08.279183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:50959)
0926 05:52:08.279317 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:08.279327 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:08.279377 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:08.279585 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:08.279585 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:08.279745 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:08.279793 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:08.279795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:08.280806 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:08.280808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:08.281412 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:08.281414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:08.281930 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:08.281931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:08.281990 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:08.282655 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:08.282662 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:08.285003 (+ 2341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
W20260926 05:52:08.285284 2317 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:08.279334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:50959 (local address 127.24.255.62:34919)
0926 05:52:08.279441 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:08.279443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:08.279454 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:52:08.279481 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:08.279482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:08.279495 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:08.279563 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:08.279947 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:08.280659 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:08.281590 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:08.281820 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:08.281896 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:08.285202 (+ 3306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:50959: BlockingRecv error: recv got EOF from 127.24.255.2:50959 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 05:52:08 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:08 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:08.400892 2319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:08.395981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:39025)
0926 05:52:08.396155 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:08.396178 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:08.396242 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:08.396377 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:08.396378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:08.396574 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:08.396628 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:08.396630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:08.397312 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:08.397313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:08.397970 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:08.397972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:08.398522 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:08.398523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:08.398580 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:08.399056 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:08.399062 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:08.400829 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":75,"threads_started":1}
W20260926 05:52:08.401058 2317 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:08.396177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:39025 (local address 127.24.255.62:34919)
0926 05:52:08.396253 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:08.396255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:08.396269 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:52:08.396302 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:08.396304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:08.396320 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:08.396367 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:08.396708 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:08.397213 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:08.398127 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:08.398439 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:08.398513 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:08.400981 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:39025: BlockingRecv error: recv got EOF from 127.24.255.3:39025 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 05:52:09 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:09 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:09.138947 2320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:09.133609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:47677)
0926 05:52:09.133733 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:09.133745 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:09.133803 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:09.134393 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:09.134394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:09.134611 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:09.134669 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:09.134671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:09.135457 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:09.135458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:09.136010 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:09.136012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:09.136489 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:09.136490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:09.136544 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:09.137011 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:09.137016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:09.138870 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
W20260926 05:52:09.139076 2321 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:09.133815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:47677 (local address 127.24.255.62:34919)
0926 05:52:09.134029 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:09.134031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:09.134046 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:52:09.134223 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:09.134225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:09.134246 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:09.134344 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:09.134752 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:09.135318 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:09.136097 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:09.136422 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:09.136476 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:09.139005 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:47677: BlockingRecv error: recv got EOF from 127.24.255.1:47677 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":126,"threads_started":1}
Sep 26 05:52:09 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:09 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:09.291747 2322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:09.285903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:53451)
0926 05:52:09.286089 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:09.286127 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:09.286187 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:09.286363 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:09.286364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:09.286534 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:09.286584 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:09.286586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:09.287274 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:09.287275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:09.287818 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:09.287821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:09.288440 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:09.288441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:09.288498 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:09.288983 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:09.288987 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:09.291668 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
W20260926 05:52:09.292014 2321 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:09.286042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:53451 (local address 127.24.255.62:34919)
0926 05:52:09.286161 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:09.286163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:09.286177 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:52:09.286252 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:09.286253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:09.286264 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:09.286321 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:09.286682 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:09.287186 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:09.287985 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:09.288306 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:09.288416 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:09.291964 (+ 3548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:53451: BlockingRecv error: recv got EOF from 127.24.255.2:53451 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 05:52:09 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:09 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:09.406906 2323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:09.401541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:47293)
0926 05:52:09.401698 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:09.401710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:09.401761 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:09.402034 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:09.402035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:09.402225 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:09.402281 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:09.402282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:09.403022 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:09.403022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:09.403545 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:09.403547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:09.403977 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:09.403978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:09.404029 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:09.404630 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:09.404635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:09.406815 (+ 2180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
W20260926 05:52:09.406982 2321 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:09.401795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:47293 (local address 127.24.255.62:34919)
0926 05:52:09.401861 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:09.401863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:09.401873 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:52:09.401900 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:09.401901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:09.401916 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:09.401981 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:09.402402 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:09.402905 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:09.403656 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:09.403864 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:09.403928 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:09.406915 (+ 2987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:47293: BlockingRecv error: recv got EOF from 127.24.255.3:47293 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 05:52:10 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:10 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:10.144532 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:10.139473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:45809)
0926 05:52:10.139584 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:10.139596 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:10.139653 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:10.139834 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:10.139835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:10.140018 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:10.140065 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:10.140066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:10.140760 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:10.140762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:10.141309 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:10.141310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:10.141759 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:10.141760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:10.141818 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:10.142357 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:10.142362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:10.144393 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
W20260926 05:52:10.144659 2325 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:10.139513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:45809 (local address 127.24.255.62:34919)
0926 05:52:10.139631 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:10.139633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:10.139642 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:52:10.139724 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:10.139725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:10.139736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:10.139807 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:10.140142 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:10.140672 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:10.141442 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:10.141671 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:10.141726 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:10.144550 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:45809: BlockingRecv error: recv got EOF from 127.24.255.1:45809 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
Sep 26 05:52:10 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:10 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:10.297189 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:10.292516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:36217)
0926 05:52:10.292702 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:10.292716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:10.292804 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:10.292953 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:10.292954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:10.293126 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:10.293170 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:10.293172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:10.293869 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:10.293870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:10.294413 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:10.294415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:10.294740 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:10.294741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:10.294794 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:10.295251 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:10.295256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:10.297109 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
W20260926 05:52:10.297268 2325 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:10.292622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:36217 (local address 127.24.255.62:34919)
0926 05:52:10.292708 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:10.292710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:10.292779 (+ 69us) server_negotiation.cc:407] Connection header received
0926 05:52:10.292875 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:10.292877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:10.292889 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:10.292935 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:10.293242 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:10.293786 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:10.294490 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:10.294677 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:10.294733 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:10.297212 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:36217: BlockingRecv error: recv got EOF from 127.24.255.2:36217 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 05:52:10 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:10 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:10.412261 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:10.407443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:33263)
0926 05:52:10.407592 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:10.407602 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:10.407652 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:10.407768 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:10.407768 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:10.407955 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:10.408008 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:10.408010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:10.408745 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:10.408746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:10.409283 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:10.409285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:10.409725 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:10.409726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:10.409778 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:10.410250 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:10.410255 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:10.412189 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
W20260926 05:52:10.412396 2325 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:10.407565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:33263 (local address 127.24.255.62:34919)
0926 05:52:10.407617 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:10.407619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:10.407640 (+ 21us) server_negotiation.cc:407] Connection header received
0926 05:52:10.407708 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:10.407709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:10.407721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:10.407756 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:10.408107 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:10.408609 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:10.409423 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:10.409627 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:10.409690 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:10.412321 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:33263: BlockingRecv error: recv got EOF from 127.24.255.3:33263 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 26 05:52:11 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:11 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:11.150455 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:11.145157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:41787)
0926 05:52:11.145295 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:11.145307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:11.145363 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:11.145648 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:11.145649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:11.145808 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:11.145853 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:11.145855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:11.146649 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:11.146650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:11.147286 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:11.147289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:11.147708 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:11.147709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:11.147768 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:11.148376 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:11.148381 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:11.150392 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
W20260926 05:52:11.150699 2329 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:11.145295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:41787 (local address 127.24.255.62:34919)
0926 05:52:11.145465 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:11.145466 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:11.145475 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:52:11.145501 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:11.145501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:11.145512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:11.145591 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:11.145956 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:11.146563 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:11.147388 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:11.147595 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:11.147658 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:11.150633 (+ 2975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:41787: BlockingRecv error: recv got EOF from 127.24.255.1:41787 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":66,"threads_started":1}
Sep 26 05:52:11 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:11 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:11.303035 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:11.297781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:58187)
0926 05:52:11.297916 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:11.297944 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:11.298006 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:11.298171 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:11.298172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:11.298342 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:11.298386 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:11.298388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:11.299020 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:11.299021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:11.299558 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:11.299560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:11.299892 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:11.299893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:11.299952 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:11.300500 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:11.300504 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:11.302931 (+ 2427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
W20260926 05:52:11.303087 2329 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:11.297951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:58187 (local address 127.24.255.62:34919)
0926 05:52:11.298024 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:11.298026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:11.298046 (+ 20us) server_negotiation.cc:407] Connection header received
0926 05:52:11.298075 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:11.298075 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:11.298086 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:11.298166 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:11.298463 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:11.298945 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:11.299647 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:11.299832 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:11.299889 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:11.303031 (+ 3142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:58187: BlockingRecv error: recv got EOF from 127.24.255.2:58187 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 05:52:11 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:11 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:11.417484 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:11.412920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:44401)
0926 05:52:11.413080 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:11.413092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:11.413175 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:11.413293 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:11.413294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:11.413461 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:11.413511 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:11.413513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:11.414194 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:11.414195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:11.414755 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:11.414757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:11.415160 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:11.415161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:11.415207 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:11.415673 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:11.415677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:11.417423 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":71,"threads_started":1}
W20260926 05:52:11.417587 2329 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:11.413060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:44401 (local address 127.24.255.62:34919)
0926 05:52:11.413112 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:11.413114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:11.413148 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:52:11.413222 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:11.413223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:11.413236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:11.413279 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:11.413568 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:11.414060 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:11.414881 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:11.415077 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:11.415141 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:11.417524 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:44401: BlockingRecv error: recv got EOF from 127.24.255.3:44401 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 05:52:12 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:12 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:12.156538 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:12.151141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:34769)
0926 05:52:12.151309 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:12.151323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:12.151479 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:12.151721 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:12.151722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:12.151909 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:12.151952 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:12.151954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:12.152798 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:12.152799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:12.153328 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:12.153329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:12.153679 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:12.153680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:12.153728 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:12.154210 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:12.154215 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:12.156395 (+ 2180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
W20260926 05:52:12.156586 2333 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:12.151264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:34769 (local address 127.24.255.62:34919)
0926 05:52:12.151424 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:12.151426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:12.151460 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:52:12.151568 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:12.151570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:12.151582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:12.151650 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:12.152050 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:12.152709 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:12.153402 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:12.153616 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:12.153671 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:12.156524 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:34769: BlockingRecv error: recv got EOF from 127.24.255.1:34769 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
Sep 26 05:52:12 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:12 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:12.314563 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:12.303759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:51077)
0926 05:52:12.303940 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:12.303951 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:12.304027 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:12.304174 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:12.304175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:12.304383 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:12.304441 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:12.304443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:12.305307 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:12.305308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:12.306202 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:12.306205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:12.306632 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:12.306633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:12.306687 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:12.312306 (+ 5619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:12.312313 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:12.314461 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":63,"threads_started":1}
W20260926 05:52:12.314630 2333 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:12.303873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:51077 (local address 127.24.255.62:34919)
0926 05:52:12.303936 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:12.303938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:12.304015 (+ 77us) server_negotiation.cc:407] Connection header received
0926 05:52:12.304097 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:12.304098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:12.304111 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:12.304163 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:12.304514 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:12.305227 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:12.306282 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:12.306567 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:12.306626 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:12.314581 (+ 7955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:51077: BlockingRecv error: recv got EOF from 127.24.255.2:51077 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 05:52:12 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:12 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.3@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:12.424619 2335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:12.418138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.3:52767)
0926 05:52:12.418326 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:12.418382 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:12.418453 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:12.418812 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:12.418813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:12.419020 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:12.419070 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:12.419072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:12.420003 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:12.420005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:12.420551 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:12.420552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:12.421007 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:12.421009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:12.421118 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:12.421999 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:12.422005 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:12.424552 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
W20260926 05:52:12.424979 2333 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:12.418252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:52767 (local address 127.24.255.62:34919)
0926 05:52:12.418319 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:12.418321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:12.418437 (+ 116us) server_negotiation.cc:407] Connection header received
0926 05:52:12.418508 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:12.418509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:12.418524 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:12.418576 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:12.419184 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:12.419709 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:12.420648 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:12.420858 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:12.420925 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:12.424905 (+ 3980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.3:52767: BlockingRecv error: recv got EOF from 127.24.255.3:52767 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 05:52:13 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:13 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.1@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:13.161960 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:13.157258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.1:58887)
0926 05:52:13.157399 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:13.157412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:13.157491 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:13.157766 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:13.157767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:13.157929 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:13.157974 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:13.157975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.158760 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.158761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:13.159297 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:13.159299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.159688 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.159689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:13.159738 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:13.160247 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:13.160252 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:13.161881 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
W20260926 05:52:13.162083 2337 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:13.157434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:58887 (local address 127.24.255.62:34919)
0926 05:52:13.157602 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:13.157603 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:13.157617 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:52:13.157644 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:13.157645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:13.157656 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:13.157725 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:13.158069 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.158697 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.159392 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.159594 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.159673 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:13.161997 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.1:58887: BlockingRecv error: recv got EOF from 127.24.255.1:58887 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
Sep 26 05:52:13 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:52:13 dist-test-slave-7fr8 krb5kdc[1278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.255.2@KRBTEST.COM for oryx/127.24.255.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:52:13.319896 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:13.315132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34919 (local address 127.24.255.2:45821)
0926 05:52:13.315259 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:13.315270 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:13.315351 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:13.315492 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:13.315493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:13.315661 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:13.315707 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:13.315708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.316370 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.316371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:13.316893 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:13.316894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.317403 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.317405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:13.317472 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:13.317977 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:13.317983 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:13.319816 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.255.62:34919: Server oryx/127.24.255.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
W20260926 05:52:13.320003 2337 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:13.315187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:45821 (local address 127.24.255.62:34919)
0926 05:52:13.315239 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:13.315241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:13.315344 (+ 103us) server_negotiation.cc:407] Connection header received
0926 05:52:13.315410 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:13.315411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:13.315422 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:13.315485 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:13.315779 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.316287 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.317021 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.317275 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.317375 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:13.319933 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.255.2:45821: BlockingRecv error: recv got EOF from 127.24.255.2:45821 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:52:13.336437 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 1861
I20260926 05:52:13.340648 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 2013
I20260926 05:52:13.344702 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 2146
I20260926 05:52:13.348512 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 1791
2026-09-26T05:52:13Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17448 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T05:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:52:13Z Disabled control of system clock
I20260926 05:52:13.375088 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:42287
--webserver_interface=127.24.255.62
--webserver_port=0
--builtin_ntp_servers=127.24.255.20:45819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:42287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 05:52:13.458782 2342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:52:13.459012 2342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:52:13.459031 2342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:52:13.460291 2342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:52:13.460333 2342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:52:13.460345 2342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:52:13.460364 2342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:52:13.460376 2342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:52:13.461737 2342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:45819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:42287
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:42287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:52:13.461946 2342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:52:13.462226 2342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:52:13.467423 2342 server_base.cc:1034] running on GCE node
W20260926 05:52:13.467422 2348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:13.467528 2347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:13.467695 2350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:13.467971 2342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:52:13.468319 2342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:52:13.469501 2342 hybrid_clock.cc:648] HybridClock initialized: now 1790401933469480 us; error 37 us; skew 500 ppm
I20260926 05:52:13.470969 2342 webserver.cc:460] Webserver started at http://127.24.255.62:41481/ using document root <none> and password file <none>
I20260926 05:52:13.471210 2342 fs_manager.cc:359] Metadata directory not provided
I20260926 05:52:13.471261 2342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:52:13.471402 2342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:52:13.472615 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "a98f70ae84e044f287a1bfafada3206f"
format_stamp: "Formatted at 2026-09-26 05:52:13 on dist-test-slave-7fr8"
server_key: "59aa56a2b50210cc0002115e3ac73f8a"
server_key_iv: "768de504696acfb35a90d79f9bbcdb5e"
server_key_version: "encryptionkey@0"
I20260926 05:52:13.472980 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "a98f70ae84e044f287a1bfafada3206f"
format_stamp: "Formatted at 2026-09-26 05:52:13 on dist-test-slave-7fr8"
server_key: "59aa56a2b50210cc0002115e3ac73f8a"
server_key_iv: "768de504696acfb35a90d79f9bbcdb5e"
server_key_version: "encryptionkey@0"
I20260926 05:52:13.474416 2342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:52:13.475237 2356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:13.475523 2342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:52:13.475612 2342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "a98f70ae84e044f287a1bfafada3206f"
format_stamp: "Formatted at 2026-09-26 05:52:13 on dist-test-slave-7fr8"
server_key: "59aa56a2b50210cc0002115e3ac73f8a"
server_key_iv: "768de504696acfb35a90d79f9bbcdb5e"
server_key_version: "encryptionkey@0"
I20260926 05:52:13.475677 2342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:52:13.501915 2342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:52:13.502444 2342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:52:13.502610 2342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:52:13.506330 2342 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:42287
I20260926 05:52:13.506374 2408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:42287 every 8 connection(s)
I20260926 05:52:13.506708 2342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:52:13.507359 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:52:13.509948 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 2342
I20260926 05:52:13.510037 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:52:13.510068 2409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f: Bootstrap starting.
I20260926 05:52:13.510249 25596 external_mini_cluster.cc:1428] Setting key 73807c889f283ae62a283b7410ed15a0
I20260926 05:52:13.510835 2409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f: Neither blocks nor log segments found. Creating new log.
I20260926 05:52:13.511164 2409 log.cc:824] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f: Log is configured to *not* fsync() on all Append() calls
I20260926 05:52:13.512130 2409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f: No bootstrap required, opened a new log
I20260926 05:52:13.513923 2409 raft_consensus.cc:348] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98f70ae84e044f287a1bfafada3206f" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42287 } }
I20260926 05:52:13.514074 2409 raft_consensus.cc:374] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:52:13.514135 2409 raft_consensus.cc:729] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98f70ae84e044f287a1bfafada3206f, State: Initialized, Role: FOLLOWER
I20260926 05:52:13.514269 2409 consensus_queue.cc:260] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98f70ae84e044f287a1bfafada3206f" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42287 } }
I20260926 05:52:13.514364 2409 raft_consensus.cc:388] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:52:13.514396 2409 raft_consensus.cc:482] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:52:13.514443 2409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:52:13.515027 2409 raft_consensus.cc:504] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98f70ae84e044f287a1bfafada3206f" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42287 } }
I20260926 05:52:13.515123 2409 leader_election.cc:302] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a98f70ae84e044f287a1bfafada3206f; no voters:
I20260926 05:52:13.515306 2409 leader_election.cc:288] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:52:13.515355 2414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:52:13.515491 2414 raft_consensus.cc:686] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [term 1 LEADER]: Becoming Leader. State: Replica: a98f70ae84e044f287a1bfafada3206f, State: Running, Role: LEADER
I20260926 05:52:13.515614 2409 sys_catalog.cc:564] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:52:13.515630 2414 consensus_queue.cc:237] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98f70ae84e044f287a1bfafada3206f" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42287 } }
I20260926 05:52:13.516004 2412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:13.510836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44188 (local address 127.24.255.62:42287)
0926 05:52:13.511081 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:13.511091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:13.511127 (+ 36us) server_negotiation.cc:407] Connection header received
0926 05:52:13.511787 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:13.511797 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:13.511882 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:13.512051 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:13.512279 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.513200 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.513954 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.514475 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.514584 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:13.515328 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:52:13.515333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:52:13.515333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:52:13.515545 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:52:13.515587 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:52:13.515603 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:52:13.515659 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:13.515730 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 05:52:13.515810 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260926 05:52:13.516060 2414 sys_catalog.cc:455] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a98f70ae84e044f287a1bfafada3206f. Latest consensus state: current_term: 1 leader_uuid: "a98f70ae84e044f287a1bfafada3206f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98f70ae84e044f287a1bfafada3206f" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42287 } } }
I20260926 05:52:13.516366 2414 sys_catalog.cc:458] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [sys.catalog]: This master's current role is: LEADER
I20260926 05:52:13.516069 2416 sys_catalog.cc:455] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a98f70ae84e044f287a1bfafada3206f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98f70ae84e044f287a1bfafada3206f" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42287 } } }
I20260926 05:52:13.516556 2416 sys_catalog.cc:458] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f [sys.catalog]: This master's current role is: LEADER
I20260926 05:52:13.516575 2423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:52:13.517038 2423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:52:13.518746 2423 catalog_manager.cc:1347] Generated new cluster ID: 8289f82485ef4fc78a5b458b8aeae735
I20260926 05:52:13.518795 2423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:52:13.522781 2423 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:52:13.523310 2423 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:52:13.526777 2423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a98f70ae84e044f287a1bfafada3206f: Generated new TSK 0
I20260926 05:52:13.526978 2423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:52:13.532383 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42287
--builtin_ntp_servers=127.24.255.20:45819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 05:52:13.614073 2433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:52:13.614359 2433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:52:13.614382 2433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:52:13.615748 2433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:52:13.615801 2433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:52:13.615839 2433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:52:13.617419 2433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:45819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:52:13.617688 2433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:52:13.617977 2433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:52:13.624152 2439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:13.624131 2438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:13.624135 2441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:13.624981 2433 server_base.cc:1034] running on GCE node
I20260926 05:52:13.625203 2433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:52:13.625541 2433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:52:13.626684 2433 hybrid_clock.cc:648] HybridClock initialized: now 1790401933626661 us; error 38 us; skew 500 ppm
I20260926 05:52:13.628091 2433 webserver.cc:460] Webserver started at http://127.24.255.1:38557/ using document root <none> and password file <none>
I20260926 05:52:13.628320 2433 fs_manager.cc:359] Metadata directory not provided
I20260926 05:52:13.628371 2433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:52:13.628501 2433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:52:13.629575 2433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "cda4209083244afe8c06f8d0127fb957"
format_stamp: "Formatted at 2026-09-26 05:52:13 on dist-test-slave-7fr8"
server_key: "919ca3e78d16135a75fc516943c9cea5"
server_key_iv: "d9a4586c25c9e4d2bfef7232980acb07"
server_key_version: "encryptionkey@0"
I20260926 05:52:13.629921 2433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "cda4209083244afe8c06f8d0127fb957"
format_stamp: "Formatted at 2026-09-26 05:52:13 on dist-test-slave-7fr8"
server_key: "919ca3e78d16135a75fc516943c9cea5"
server_key_iv: "d9a4586c25c9e4d2bfef7232980acb07"
server_key_version: "encryptionkey@0"
I20260926 05:52:13.631290 2433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:52:13.632037 2447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:13.632195 2433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:52:13.632278 2433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "cda4209083244afe8c06f8d0127fb957"
format_stamp: "Formatted at 2026-09-26 05:52:13 on dist-test-slave-7fr8"
server_key: "919ca3e78d16135a75fc516943c9cea5"
server_key_iv: "d9a4586c25c9e4d2bfef7232980acb07"
server_key_version: "encryptionkey@0"
I20260926 05:52:13.632354 2433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:52:13.642850 2433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:52:13.643222 2433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:52:13.643352 2433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:52:13.643577 2433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:52:13.643901 2433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:52:13.643936 2433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:13.643965 2433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:52:13.643982 2433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:13.649443 2433 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:42561
I20260926 05:52:13.649521 2560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:42561 every 8 connection(s)
I20260926 05:52:13.649813 2433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:52:13.655025 2412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:13.650582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:43239 (local address 127.24.255.62:42287)
0926 05:52:13.650662 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:13.650664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:13.650828 (+ 164us) server_negotiation.cc:407] Connection header received
0926 05:52:13.651319 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:13.651321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:13.651331 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:13.651434 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:13.652061 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.652590 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.653611 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.653824 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.653895 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:13.654714 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:52:13.654715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:52:13.654716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:52:13.654722 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:52:13.654739 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:52:13.654744 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:52:13.654806 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:13.654931 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 05:52:13.654957 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:52:13.655236 2563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:13.650509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:42287 (local address 127.24.255.1:43239)
0926 05:52:13.650685 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:13.650723 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:13.651222 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:13.651495 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:13.651498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:13.651506 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:13.651985 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:13.651989 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.652682 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.652683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:13.653502 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:13.653504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.653899 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.653900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:13.654019 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:13.654587 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:52:13.654587 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:13.654614 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:52:13.654638 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:52:13.654843 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:52:13.654855 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:52:13.654858 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:52:13.654928 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 05:52:13.655018 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260926 05:52:13.655800 2561 heartbeater.cc:347] Connected to a master server at 127.24.255.62:42287
I20260926 05:52:13.655910 2561 heartbeater.cc:464] Registering TS with master...
I20260926 05:52:13.656097 2561 heartbeater.cc:511] Master 127.24.255.62:42287 requested a full tablet report, sending...
I20260926 05:52:13.656800 2373 ts_manager.cc:194] Registered new tserver with Master: cda4209083244afe8c06f8d0127fb957 (127.24.255.1:42561)
I20260926 05:52:13.657213 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 2433
I20260926 05:52:13.657289 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:52:13.657464 2373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.255.1:43239
I20260926 05:52:13.657521 25596 external_mini_cluster.cc:1428] Setting key bbb689cda73c39705fd67b4369e3e48f
I20260926 05:52:13.658388 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42287
--builtin_ntp_servers=127.24.255.20:45819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 05:52:13.743672 2564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:52:13.743899 2564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:52:13.743922 2564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:52:13.745299 2564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:52:13.745366 2564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:52:13.745406 2564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:52:13.746923 2564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:45819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:52:13.747198 2564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:52:13.747665 2564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:52:13.752887 2569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:13.752938 2570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:13.752887 2572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:13.753274 2564 server_base.cc:1034] running on GCE node
I20260926 05:52:13.753542 2564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:52:13.753782 2564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:52:13.754949 2564 hybrid_clock.cc:648] HybridClock initialized: now 1790401933754875 us; error 89 us; skew 500 ppm
I20260926 05:52:13.756342 2564 webserver.cc:460] Webserver started at http://127.24.255.2:41765/ using document root <none> and password file <none>
I20260926 05:52:13.756577 2564 fs_manager.cc:359] Metadata directory not provided
I20260926 05:52:13.756629 2564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:52:13.756753 2564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:52:13.757998 2564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "b62eca99e2d740258b831519c5763477"
format_stamp: "Formatted at 2026-09-26 05:52:13 on dist-test-slave-7fr8"
server_key: "7a6e17c8342b481a49e309c89e926a01"
server_key_iv: "f8dca8edb298d4289be88c2d3d39cf8d"
server_key_version: "encryptionkey@0"
I20260926 05:52:13.758431 2564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "b62eca99e2d740258b831519c5763477"
format_stamp: "Formatted at 2026-09-26 05:52:13 on dist-test-slave-7fr8"
server_key: "7a6e17c8342b481a49e309c89e926a01"
server_key_iv: "f8dca8edb298d4289be88c2d3d39cf8d"
server_key_version: "encryptionkey@0"
I20260926 05:52:13.759771 2564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:52:13.760644 2578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:13.760828 2564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:52:13.760911 2564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "b62eca99e2d740258b831519c5763477"
format_stamp: "Formatted at 2026-09-26 05:52:13 on dist-test-slave-7fr8"
server_key: "7a6e17c8342b481a49e309c89e926a01"
server_key_iv: "f8dca8edb298d4289be88c2d3d39cf8d"
server_key_version: "encryptionkey@0"
I20260926 05:52:13.760977 2564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:52:13.769399 2564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:52:13.769713 2564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:52:13.770335 2564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:52:13.770641 2564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:52:13.770999 2564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:52:13.771037 2564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:13.771066 2564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:52:13.771085 2564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:13.776721 2564 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:35883
I20260926 05:52:13.776767 2691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:35883 every 8 connection(s)
I20260926 05:52:13.777101 2564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:52:13.782601 2412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:13.777844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:51479 (local address 127.24.255.62:42287)
0926 05:52:13.777915 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:13.777917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:13.778182 (+ 265us) server_negotiation.cc:407] Connection header received
0926 05:52:13.778756 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:13.778758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:13.778769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:13.778826 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:13.779567 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.780067 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.781099 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.781292 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.781416 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:13.782385 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:52:13.782386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:52:13.782386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:52:13.782391 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:52:13.782407 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:52:13.782411 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:52:13.782449 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:13.782512 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 05:52:13.782537 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:52:13.782790 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:13.777801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:42287 (local address 127.24.255.2:51479)
0926 05:52:13.778039 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:13.778083 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:13.778668 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:13.778899 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:13.778902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:13.778912 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:13.779474 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:13.779480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.780164 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.780166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:13.781005 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:13.781007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.781452 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.781453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:13.781579 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:13.782228 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:52:13.782228 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:13.782259 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:52:13.782292 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:52:13.782455 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:52:13.782458 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:52:13.782460 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:52:13.782509 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 05:52:13.782593 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
I20260926 05:52:13.783219 2692 heartbeater.cc:347] Connected to a master server at 127.24.255.62:42287
I20260926 05:52:13.783303 2692 heartbeater.cc:464] Registering TS with master...
I20260926 05:52:13.783309 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 2564
I20260926 05:52:13.783401 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:52:13.783494 2692 heartbeater.cc:511] Master 127.24.255.62:42287 requested a full tablet report, sending...
I20260926 05:52:13.783638 25596 external_mini_cluster.cc:1428] Setting key 50443de21e01623063c923e2b4b8402b
I20260926 05:52:13.783864 2373 ts_manager.cc:194] Registered new tserver with Master: b62eca99e2d740258b831519c5763477 (127.24.255.2:35883)
I20260926 05:52:13.784435 2373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.255.2:51479
I20260926 05:52:13.784746 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42287
--builtin_ntp_servers=127.24.255.20:45819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 05:52:13.869620 2695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:52:13.869838 2695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:52:13.869860 2695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:52:13.871129 2695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:52:13.871177 2695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:52:13.871203 2695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:52:13.872623 2695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:45819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:52:13.872869 2695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:52:13.873212 2695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:52:13.878329 2701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:13.878347 2703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:13.878351 2700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:13.878696 2695 server_base.cc:1034] running on GCE node
I20260926 05:52:13.878981 2695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:52:13.879320 2695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:52:13.880496 2695 hybrid_clock.cc:648] HybridClock initialized: now 1790401933880482 us; error 29 us; skew 500 ppm
I20260926 05:52:13.881868 2695 webserver.cc:460] Webserver started at http://127.24.255.3:35405/ using document root <none> and password file <none>
I20260926 05:52:13.882130 2695 fs_manager.cc:359] Metadata directory not provided
I20260926 05:52:13.882189 2695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:52:13.882308 2695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:52:13.883416 2695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "9e98a6d5135443ba8d58d0ae5cf38e17"
format_stamp: "Formatted at 2026-09-26 05:52:13 on dist-test-slave-7fr8"
server_key: "e671243c249f8f35b2ae4b9df91d3757"
server_key_iv: "dc076c44a8fe46d79ffe4eec1105d64a"
server_key_version: "encryptionkey@0"
I20260926 05:52:13.883754 2695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "9e98a6d5135443ba8d58d0ae5cf38e17"
format_stamp: "Formatted at 2026-09-26 05:52:13 on dist-test-slave-7fr8"
server_key: "e671243c249f8f35b2ae4b9df91d3757"
server_key_iv: "dc076c44a8fe46d79ffe4eec1105d64a"
server_key_version: "encryptionkey@0"
I20260926 05:52:13.885182 2695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:52:13.885965 2709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:13.886173 2695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:52:13.886258 2695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "9e98a6d5135443ba8d58d0ae5cf38e17"
format_stamp: "Formatted at 2026-09-26 05:52:13 on dist-test-slave-7fr8"
server_key: "e671243c249f8f35b2ae4b9df91d3757"
server_key_iv: "dc076c44a8fe46d79ffe4eec1105d64a"
server_key_version: "encryptionkey@0"
I20260926 05:52:13.886332 2695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:52:13.898729 2695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:52:13.899057 2695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:52:13.899186 2695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:52:13.899401 2695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:52:13.899721 2695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:52:13.899755 2695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:13.899784 2695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:52:13.899809 2695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:13.905737 2695 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:44453
I20260926 05:52:13.905973 2822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:44453 every 8 connection(s)
I20260926 05:52:13.906248 2695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:52:13.909745 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 2695
I20260926 05:52:13.909857 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:52:13.910046 25596 external_mini_cluster.cc:1428] Setting key cc5b0e160eb5a51f988461b7d3371d7d
I20260926 05:52:13.912042 2412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:13.908008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:50439 (local address 127.24.255.62:42287)
0926 05:52:13.908130 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:13.908132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:13.908142 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:52:13.908170 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:13.908171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:13.908182 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:13.908234 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:13.908957 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.909461 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.910594 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.910826 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.910893 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:13.911730 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:52:13.911732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:52:13.911732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:52:13.911737 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:52:13.911753 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:52:13.911757 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:52:13.911835 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:13.911950 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 05:52:13.911975 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 05:52:13.912305 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:13.907074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:42287 (local address 127.24.255.3:50439)
0926 05:52:13.907263 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:13.907305 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:13.907791 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:13.908322 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:13.908326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:13.908335 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:13.908856 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:13.908860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.909563 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.909564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:13.910518 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:13.910522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:13.910933 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:13.910934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:13.911119 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:13.911585 (+ 466us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:52:13.911585 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:13.911614 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:52:13.911637 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:52:13.911865 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:52:13.911871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:52:13.911874 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:52:13.911951 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 05:52:13.912057 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260926 05:52:13.913050 2823 heartbeater.cc:347] Connected to a master server at 127.24.255.62:42287
I20260926 05:52:13.913158 2823 heartbeater.cc:464] Registering TS with master...
I20260926 05:52:13.913359 2823 heartbeater.cc:511] Master 127.24.255.62:42287 requested a full tablet report, sending...
I20260926 05:52:13.913851 2373 ts_manager.cc:194] Registered new tserver with Master: 9e98a6d5135443ba8d58d0ae5cf38e17 (127.24.255.3:44453)
I20260926 05:52:13.914578 2373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.255.3:50439
I20260926 05:52:13.922466 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 05:52:13.924358 2412 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:13.923947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.24.255.62:42287)
0926 05:52:13.924000 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:13.924002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:13.924055 (+ 53us) server_negotiation.cc:407] Connection header received
0926 05:52:13.924108 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:13.924109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:13.924118 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:13.924169 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:13.924282 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44202: BlockingRecv error: recv got EOF from 127.0.0.1:44202 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:52:13.924477 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 2433
I20260926 05:52:13.928702 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 2564
I20260926 05:52:13.932739 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 2695
I20260926 05:52:13.936631 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 2342
2026-09-26T05:52:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (586 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T05:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:52:13Z Disabled control of system clock
I20260926 05:52:13.962225 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:46073
--webserver_interface=127.24.255.62
--webserver_port=0
--builtin_ntp_servers=127.24.255.20:36561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:46073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 05:52:14.049153 2834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:52:14.049383 2834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:52:14.049405 2834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:52:14.050706 2834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:52:14.050765 2834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:52:14.050776 2834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:52:14.050787 2834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:52:14.050798 2834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:52:14.052520 2834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:46073
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:46073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:52:14.052786 2834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:52:14.053062 2834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:52:14.058141 2839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:14.058275 2834 server_base.cc:1034] running on GCE node
W20260926 05:52:14.058141 2840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:14.058321 2842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:14.058719 2834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:52:14.059051 2834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:52:14.060225 2834 hybrid_clock.cc:648] HybridClock initialized: now 1790401934060204 us; error 33 us; skew 500 ppm
I20260926 05:52:14.061640 2834 webserver.cc:460] Webserver started at http://127.24.255.62:38815/ using document root <none> and password file <none>
I20260926 05:52:14.061884 2834 fs_manager.cc:359] Metadata directory not provided
I20260926 05:52:14.061935 2834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:52:14.062083 2834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:52:14.063195 2834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "0594253e715848c7ad289cb6059d8ddc"
format_stamp: "Formatted at 2026-09-26 05:52:14 on dist-test-slave-7fr8"
server_key: "13682d0b9c691006f427ee99270b6ec2"
server_key_iv: "c36db31b98f900fa0d42f705252657c9"
server_key_version: "encryptionkey@0"
I20260926 05:52:14.063525 2834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "0594253e715848c7ad289cb6059d8ddc"
format_stamp: "Formatted at 2026-09-26 05:52:14 on dist-test-slave-7fr8"
server_key: "13682d0b9c691006f427ee99270b6ec2"
server_key_iv: "c36db31b98f900fa0d42f705252657c9"
server_key_version: "encryptionkey@0"
I20260926 05:52:14.064818 2834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:52:14.065595 2848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:14.065747 2834 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:52:14.065831 2834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "0594253e715848c7ad289cb6059d8ddc"
format_stamp: "Formatted at 2026-09-26 05:52:14 on dist-test-slave-7fr8"
server_key: "13682d0b9c691006f427ee99270b6ec2"
server_key_iv: "c36db31b98f900fa0d42f705252657c9"
server_key_version: "encryptionkey@0"
I20260926 05:52:14.065899 2834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:52:14.069317 2834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:52:14.069602 2834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:52:14.069727 2834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:52:14.073529 2834 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:46073
I20260926 05:52:14.073661 2900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:46073 every 8 connection(s)
I20260926 05:52:14.073920 2834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:52:14.074654 2901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:52:14.076977 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 2834
I20260926 05:52:14.077081 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:52:14.077265 25596 external_mini_cluster.cc:1428] Setting key 39420721b6433a2cde0dc4b30d2144e8
I20260926 05:52:14.077765 2901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc: Bootstrap starting.
I20260926 05:52:14.078522 2901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc: Neither blocks nor log segments found. Creating new log.
I20260926 05:52:14.078847 2901 log.cc:824] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc: Log is configured to *not* fsync() on all Append() calls
I20260926 05:52:14.079653 2901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc: No bootstrap required, opened a new log
I20260926 05:52:14.080317 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:14.077810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.24.255.62:46073)
0926 05:52:14.078001 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:14.078010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:14.078042 (+ 32us) server_negotiation.cc:407] Connection header received
0926 05:52:14.078661 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:14.078672 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:14.078735 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:14.078894 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:14.079583 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:52:14.079587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:52:14.079587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:52:14.079592 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:52:14.079638 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:52:14.079653 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:52:14.079743 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:14.079867 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 05:52:14.080004 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260926 05:52:14.081782 2901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0594253e715848c7ad289cb6059d8ddc" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46073 } }
I20260926 05:52:14.081928 2901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:52:14.081947 2901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0594253e715848c7ad289cb6059d8ddc, State: Initialized, Role: FOLLOWER
I20260926 05:52:14.082042 2901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0594253e715848c7ad289cb6059d8ddc" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46073 } }
I20260926 05:52:14.082149 2901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:52:14.082188 2901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:52:14.082223 2901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:52:14.082867 2901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0594253e715848c7ad289cb6059d8ddc" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46073 } }
I20260926 05:52:14.082958 2901 leader_election.cc:302] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0594253e715848c7ad289cb6059d8ddc; no voters:
I20260926 05:52:14.083115 2901 leader_election.cc:288] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:52:14.083165 2906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:52:14.083335 2906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [term 1 LEADER]: Becoming Leader. State: Replica: 0594253e715848c7ad289cb6059d8ddc, State: Running, Role: LEADER
I20260926 05:52:14.083372 2901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:52:14.083436 2906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0594253e715848c7ad289cb6059d8ddc" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46073 } }
I20260926 05:52:14.083806 2907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0594253e715848c7ad289cb6059d8ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0594253e715848c7ad289cb6059d8ddc" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46073 } } }
I20260926 05:52:14.083910 2907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [sys.catalog]: This master's current role is: LEADER
I20260926 05:52:14.083799 2908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0594253e715848c7ad289cb6059d8ddc. Latest consensus state: current_term: 1 leader_uuid: "0594253e715848c7ad289cb6059d8ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0594253e715848c7ad289cb6059d8ddc" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46073 } } }
I20260926 05:52:14.084052 2908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc [sys.catalog]: This master's current role is: LEADER
W20260926 05:52:14.085085 2922 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:52:14.085163 2922 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:52:14.085227 2915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:52:14.085412 2915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:52:14.086962 2915 catalog_manager.cc:1347] Generated new cluster ID: e5518d4b5ac849f6ab9d27801b479f56
I20260926 05:52:14.087014 2915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:52:14.106561 2915 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:52:14.106752 2915 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:52:14.110335 2915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0594253e715848c7ad289cb6059d8ddc: Generated new TSK 0
I20260926 05:52:14.110550 2915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:52:14.112493 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:46073
--builtin_ntp_servers=127.24.255.20:36561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 05:52:14.215756 2925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:52:14.215981 2925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:52:14.216006 2925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:52:14.217293 2925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:52:14.217341 2925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:52:14.217370 2925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:52:14.219026 2925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:46073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:52:14.219306 2925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:52:14.219609 2925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:52:14.224974 2933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:14.224961 2930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:14.224961 2931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:14.225502 2925 server_base.cc:1034] running on GCE node
I20260926 05:52:14.225703 2925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:52:14.225955 2925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:52:14.227125 2925 hybrid_clock.cc:648] HybridClock initialized: now 1790401934227110 us; error 33 us; skew 500 ppm
I20260926 05:52:14.228425 2925 webserver.cc:460] Webserver started at http://127.24.255.1:45321/ using document root <none> and password file <none>
I20260926 05:52:14.228688 2925 fs_manager.cc:359] Metadata directory not provided
I20260926 05:52:14.228744 2925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:52:14.228878 2925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:52:14.229959 2925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "0c3147ca33ee412e889c349ea3b955ca"
format_stamp: "Formatted at 2026-09-26 05:52:14 on dist-test-slave-7fr8"
server_key: "3c58807aa1fa7875f4c2fe1294f5008c"
server_key_iv: "5e2852759b4ad1f5b2abec1f408f925f"
server_key_version: "encryptionkey@0"
I20260926 05:52:14.230322 2925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "0c3147ca33ee412e889c349ea3b955ca"
format_stamp: "Formatted at 2026-09-26 05:52:14 on dist-test-slave-7fr8"
server_key: "3c58807aa1fa7875f4c2fe1294f5008c"
server_key_iv: "5e2852759b4ad1f5b2abec1f408f925f"
server_key_version: "encryptionkey@0"
I20260926 05:52:14.231668 2925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:52:14.232479 2939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:14.232658 2925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:52:14.232748 2925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "0c3147ca33ee412e889c349ea3b955ca"
format_stamp: "Formatted at 2026-09-26 05:52:14 on dist-test-slave-7fr8"
server_key: "3c58807aa1fa7875f4c2fe1294f5008c"
server_key_iv: "5e2852759b4ad1f5b2abec1f408f925f"
server_key_version: "encryptionkey@0"
I20260926 05:52:14.232831 2925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:52:14.236523 2925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:52:14.236821 2925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:52:14.236959 2925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:52:14.237169 2925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:52:14.237490 2925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:52:14.237528 2925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:14.237562 2925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:52:14.237591 2925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:14.243412 2925 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:37225
I20260926 05:52:14.243456 3052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:37225 every 8 connection(s)
I20260926 05:52:14.243814 2925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:52:14.246917 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:14.244544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:59835 (local address 127.24.255.62:46073)
0926 05:52:14.244596 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:14.244598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:14.244821 (+ 223us) server_negotiation.cc:407] Connection header received
0926 05:52:14.245320 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:14.245322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:14.245328 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:14.245377 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:14.246673 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:52:14.246675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:52:14.246675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:52:14.246680 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:52:14.246697 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:52:14.246701 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:52:14.246751 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:14.246828 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 05:52:14.246843 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 05:52:14.247202 3055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:14.244530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46073 (local address 127.24.255.1:59835)
0926 05:52:14.244705 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:14.244740 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:14.245225 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:14.245455 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:14.245459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:14.245470 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:14.246491 (+ 1021us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:52:14.246492 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:14.246530 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:52:14.246565 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:52:14.246756 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:52:14.246761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:52:14.246765 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:52:14.246837 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:52:14.246902 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260926 05:52:14.247747 3053 heartbeater.cc:347] Connected to a master server at 127.24.255.62:46073
I20260926 05:52:14.247862 3053 heartbeater.cc:464] Registering TS with master...
I20260926 05:52:14.248025 3053 heartbeater.cc:511] Master 127.24.255.62:46073 requested a full tablet report, sending...
I20260926 05:52:14.248452 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 2925
I20260926 05:52:14.248540 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:52:14.248620 2865 ts_manager.cc:194] Registered new tserver with Master: 0c3147ca33ee412e889c349ea3b955ca (127.24.255.1:37225)
I20260926 05:52:14.248735 25596 external_mini_cluster.cc:1428] Setting key 1672aa508bd0525fdee8d438bedf2aa6
I20260926 05:52:14.249585 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:46073
--builtin_ntp_servers=127.24.255.20:36561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 05:52:14.333942 3056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:52:14.334235 3056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:52:14.334264 3056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:52:14.335606 3056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:52:14.335657 3056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:52:14.335687 3056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:52:14.337154 3056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:46073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:52:14.337409 3056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:52:14.337699 3056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:52:14.342713 3064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:14.342878 3056 server_base.cc:1034] running on GCE node
W20260926 05:52:14.342713 3061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:14.342713 3062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:14.343201 3056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:52:14.343432 3056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:52:14.344595 3056 hybrid_clock.cc:648] HybridClock initialized: now 1790401934344580 us; error 32 us; skew 500 ppm
I20260926 05:52:14.345851 3056 webserver.cc:460] Webserver started at http://127.24.255.2:37245/ using document root <none> and password file <none>
I20260926 05:52:14.346076 3056 fs_manager.cc:359] Metadata directory not provided
I20260926 05:52:14.346153 3056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:52:14.346279 3056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:52:14.347399 3056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "bd49a7b66a4d4530834b49ff636b7793"
format_stamp: "Formatted at 2026-09-26 05:52:14 on dist-test-slave-7fr8"
server_key: "c843ffcd2ed0dacd4d94d42f70eda186"
server_key_iv: "5fbaf0ea1fbc8a6bd072459fbcb2dcba"
server_key_version: "encryptionkey@0"
I20260926 05:52:14.347733 3056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "bd49a7b66a4d4530834b49ff636b7793"
format_stamp: "Formatted at 2026-09-26 05:52:14 on dist-test-slave-7fr8"
server_key: "c843ffcd2ed0dacd4d94d42f70eda186"
server_key_iv: "5fbaf0ea1fbc8a6bd072459fbcb2dcba"
server_key_version: "encryptionkey@0"
I20260926 05:52:14.349053 3056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:52:14.349942 3070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:14.350299 3056 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:52:14.350391 3056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "bd49a7b66a4d4530834b49ff636b7793"
format_stamp: "Formatted at 2026-09-26 05:52:14 on dist-test-slave-7fr8"
server_key: "c843ffcd2ed0dacd4d94d42f70eda186"
server_key_iv: "5fbaf0ea1fbc8a6bd072459fbcb2dcba"
server_key_version: "encryptionkey@0"
I20260926 05:52:14.350462 3056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:52:14.354048 3056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:52:14.354336 3056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:52:14.354518 3056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:52:14.354717 3056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:52:14.355026 3056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:52:14.355059 3056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:14.355087 3056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:52:14.355113 3056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:14.360697 3056 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:42911
I20260926 05:52:14.360733 3183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:42911 every 8 connection(s)
I20260926 05:52:14.361073 3056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:52:14.364310 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 3056
I20260926 05:52:14.364401 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:52:14.364583 25596 external_mini_cluster.cc:1428] Setting key e269d5e704faf0e767befe055ac78bac
I20260926 05:52:14.364672 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:14.361832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:58667 (local address 127.24.255.62:46073)
0926 05:52:14.361917 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:14.361919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:14.362249 (+ 330us) server_negotiation.cc:407] Connection header received
0926 05:52:14.362746 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:14.362747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:14.362754 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:14.362805 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:14.364373 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:52:14.364375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:52:14.364375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:52:14.364380 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:52:14.364398 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:52:14.364404 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:52:14.364453 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:14.364579 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 05:52:14.364596 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 05:52:14.364871 3186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:14.361810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46073 (local address 127.24.255.2:58667)
0926 05:52:14.362081 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:14.362155 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:14.362651 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:14.362949 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:14.362953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:14.362965 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:14.364194 (+ 1229us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:52:14.364194 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:14.364226 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:52:14.364257 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:52:14.364477 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:52:14.364482 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:52:14.364484 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:52:14.364590 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 05:52:14.364661 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260926 05:52:14.365449 3184 heartbeater.cc:347] Connected to a master server at 127.24.255.62:46073
I20260926 05:52:14.365535 3184 heartbeater.cc:464] Registering TS with master...
I20260926 05:52:14.365579 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:46073
--builtin_ntp_servers=127.24.255.20:36561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 05:52:14.365674 3184 heartbeater.cc:511] Master 127.24.255.62:46073 requested a full tablet report, sending...
I20260926 05:52:14.366019 2865 ts_manager.cc:194] Registered new tserver with Master: bd49a7b66a4d4530834b49ff636b7793 (127.24.255.2:42911)
W20260926 05:52:14.447870 3187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:52:14.448118 3187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:52:14.448143 3187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:52:14.449371 3187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:52:14.449421 3187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:52:14.449450 3187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:52:14.450889 3187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:46073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:52:14.451129 3187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:52:14.451418 3187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:52:14.456454 3193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:14.456522 3187 server_base.cc:1034] running on GCE node
W20260926 05:52:14.456470 3192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:14.456410 3195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:14.456868 3187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:52:14.457098 3187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:52:14.458233 3187 hybrid_clock.cc:648] HybridClock initialized: now 1790401934458218 us; error 32 us; skew 500 ppm
I20260926 05:52:14.459499 3187 webserver.cc:460] Webserver started at http://127.24.255.3:36085/ using document root <none> and password file <none>
I20260926 05:52:14.459729 3187 fs_manager.cc:359] Metadata directory not provided
I20260926 05:52:14.459779 3187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:52:14.459893 3187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:52:14.461030 3187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "7391c3968cb6460bbccd4424a21dcdac"
format_stamp: "Formatted at 2026-09-26 05:52:14 on dist-test-slave-7fr8"
server_key: "5cb178258ddbbdd16528feb3e1e5c701"
server_key_iv: "37a7c3caae01bfd54950489567434ecd"
server_key_version: "encryptionkey@0"
I20260926 05:52:14.461392 3187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "7391c3968cb6460bbccd4424a21dcdac"
format_stamp: "Formatted at 2026-09-26 05:52:14 on dist-test-slave-7fr8"
server_key: "5cb178258ddbbdd16528feb3e1e5c701"
server_key_iv: "37a7c3caae01bfd54950489567434ecd"
server_key_version: "encryptionkey@0"
I20260926 05:52:14.462767 3187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:52:14.463517 3201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:14.463678 3187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:52:14.463758 3187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "7391c3968cb6460bbccd4424a21dcdac"
format_stamp: "Formatted at 2026-09-26 05:52:14 on dist-test-slave-7fr8"
server_key: "5cb178258ddbbdd16528feb3e1e5c701"
server_key_iv: "37a7c3caae01bfd54950489567434ecd"
server_key_version: "encryptionkey@0"
I20260926 05:52:14.463831 3187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:52:14.467250 3187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:52:14.467514 3187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:52:14.467628 3187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:52:14.467828 3187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:52:14.468148 3187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:52:14.468183 3187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:14.468211 3187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:52:14.468227 3187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:14.473960 3187 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:36449
I20260926 05:52:14.474006 3314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:36449 every 8 connection(s)
I20260926 05:52:14.474437 3187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:52:14.477319 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:14.475187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:48971 (local address 127.24.255.62:46073)
0926 05:52:14.475272 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:14.475275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:14.475501 (+ 226us) server_negotiation.cc:407] Connection header received
0926 05:52:14.475932 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:14.475934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:14.475940 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:14.475994 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:14.477120 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:52:14.477121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:52:14.477122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:52:14.477126 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:52:14.477141 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:52:14.477145 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:52:14.477191 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:14.477249 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 05:52:14.477261 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 05:52:14.477512 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:14.475123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:46073 (local address 127.24.255.3:48971)
0926 05:52:14.475300 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:14.475375 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:14.475832 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:14.476058 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:14.476062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:14.476070 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:14.476959 (+ 889us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:52:14.476959 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:14.476994 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:52:14.477025 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:52:14.477192 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:52:14.477194 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:52:14.477197 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:52:14.477248 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:52:14.477301 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260926 05:52:14.477978 3315 heartbeater.cc:347] Connected to a master server at 127.24.255.62:46073
I20260926 05:52:14.478083 3315 heartbeater.cc:464] Registering TS with master...
I20260926 05:52:14.478292 3315 heartbeater.cc:511] Master 127.24.255.62:46073 requested a full tablet report, sending...
I20260926 05:52:14.478641 2865 ts_manager.cc:194] Registered new tserver with Master: 7391c3968cb6460bbccd4424a21dcdac (127.24.255.3:36449)
I20260926 05:52:14.480448 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 3187
I20260926 05:52:14.480520 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:52:14.480674 25596 external_mini_cluster.cc:1428] Setting key 769b520fa7f197fb4f02d499cbcfed2b
I20260926 05:52:14.481457 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 05:52:14.483239 2904 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:52:14.482775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.24.255.62:46073)
0926 05:52:14.482829 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:14.482831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:14.482946 (+ 115us) server_negotiation.cc:407] Connection header received
0926 05:52:14.482970 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:14.482971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:14.482976 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:14.483047 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:14.483157 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43658: BlockingRecv error: recv got EOF from 127.0.0.1:43658 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:52:14.483281 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 2925
I20260926 05:52:14.487828 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 3056
I20260926 05:52:14.492086 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 3187
I20260926 05:52:14.495787 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 2834
2026-09-26T05:52:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (559 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:52:14 dist-test-slave-7fr8 krb5kdc[3325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:52:14 dist-test-slave-7fr8 krb5kdc[3325](info): set up 2 sockets
Sep 26 05:52:14 dist-test-slave-7fr8 krb5kdc[3325](info): commencing operation
krb5kdc: starting...
W20260926 05:52:15.479128 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:52:15 dist-test-slave-7fr8 krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:52:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:52:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:52:15.601222 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:45279
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:45279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:52:15.684679 3338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:52:15.684916 3338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:52:15.684948 3338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:52:15.686229 3338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:52:15.686280 3338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:52:15.686297 3338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:52:15.686316 3338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:52:15.686331 3338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:52:15.687930 3338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:45279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:45279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:52:15.688230 3338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:52:15.688457 3338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:52:15.693665 3344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:15.693699 3346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:15.693876 3338 server_base.cc:1034] running on GCE node
W20260926 05:52:15.693849 3343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:15.694262 3338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:52:15.694576 3338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:52:15.695736 3338 hybrid_clock.cc:648] HybridClock initialized: now 1790401935695725 us; error 29 us; skew 500 ppm
Sep 26 05:52:15 dist-test-slave-7fr8 krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401935, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:52:15.698845 3338 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:52:15.699317 3338 webserver.cc:460] Webserver started at http://127.24.255.62:35389/ using document root <none> and password file <none>
I20260926 05:52:15.699553 3338 fs_manager.cc:359] Metadata directory not provided
I20260926 05:52:15.699604 3338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:52:15.699738 3338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:52:15.700798 3338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "01166a55c66642fc845b5c9dd380f7e1"
format_stamp: "Formatted at 2026-09-26 05:52:15 on dist-test-slave-7fr8"
server_key: "cd70f94c3211b54f8d1e4e56f01d2509"
server_key_iv: "af3287d65b8f4d089f6ec4db3bb451e9"
server_key_version: "encryptionkey@0"
I20260926 05:52:15.701130 3338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "01166a55c66642fc845b5c9dd380f7e1"
format_stamp: "Formatted at 2026-09-26 05:52:15 on dist-test-slave-7fr8"
server_key: "cd70f94c3211b54f8d1e4e56f01d2509"
server_key_iv: "af3287d65b8f4d089f6ec4db3bb451e9"
server_key_version: "encryptionkey@0"
I20260926 05:52:15.702478 3338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:52:15.703310 3353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:15.703545 3338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:52:15.703637 3338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "01166a55c66642fc845b5c9dd380f7e1"
format_stamp: "Formatted at 2026-09-26 05:52:15 on dist-test-slave-7fr8"
server_key: "cd70f94c3211b54f8d1e4e56f01d2509"
server_key_iv: "af3287d65b8f4d089f6ec4db3bb451e9"
server_key_version: "encryptionkey@0"
I20260926 05:52:15.703706 3338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:52:15.714596 3338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:52:15.716837 3338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:52:15.716997 3338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:52:15.720851 3338 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:45279
I20260926 05:52:15.720893 3405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:45279 every 8 connection(s)
I20260926 05:52:15.721240 3338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:52:15.722259 3406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:52:15.724674 3406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1: Bootstrap starting.
I20260926 05:52:15.725309 3406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1: Neither blocks nor log segments found. Creating new log.
I20260926 05:52:15.725586 3406 log.cc:824] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1: Log is configured to *not* fsync() on all Append() calls
I20260926 05:52:15.726318 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 3338
I20260926 05:52:15.726541 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:52:15.726814 25596 external_mini_cluster.cc:1428] Setting key e75ad366183b9f65a734647cda370f23
I20260926 05:52:15.726967 3406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1: No bootstrap required, opened a new log
I20260926 05:52:15.729674 3406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01166a55c66642fc845b5c9dd380f7e1" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45279 } }
I20260926 05:52:15.729859 3406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:52:15.729895 3406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01166a55c66642fc845b5c9dd380f7e1, State: Initialized, Role: FOLLOWER
I20260926 05:52:15.730064 3406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01166a55c66642fc845b5c9dd380f7e1" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45279 } }
I20260926 05:52:15.730197 3406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:52:15.730242 3406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:52:15.730297 3406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:52:15.731256 3406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01166a55c66642fc845b5c9dd380f7e1" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45279 } }
I20260926 05:52:15.731550 3406 leader_election.cc:302] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01166a55c66642fc845b5c9dd380f7e1; no voters:
I20260926 05:52:15.731846 3406 leader_election.cc:288] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:52:15.731976 3411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:52:15.732208 3411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [term 1 LEADER]: Becoming Leader. State: Replica: 01166a55c66642fc845b5c9dd380f7e1, State: Running, Role: LEADER
I20260926 05:52:15.732488 3406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:52:15.732506 3411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01166a55c66642fc845b5c9dd380f7e1" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45279 } }
I20260926 05:52:15.733022 3413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01166a55c66642fc845b5c9dd380f7e1. Latest consensus state: current_term: 1 leader_uuid: "01166a55c66642fc845b5c9dd380f7e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01166a55c66642fc845b5c9dd380f7e1" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45279 } } }
I20260926 05:52:15.733031 3412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01166a55c66642fc845b5c9dd380f7e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01166a55c66642fc845b5c9dd380f7e1" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45279 } } }
I20260926 05:52:15.733132 3412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [sys.catalog]: This master's current role is: LEADER
I20260926 05:52:15.733287 3413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1 [sys.catalog]: This master's current role is: LEADER
I20260926 05:52:15.734405 3420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:52:15.734872 3420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:52:15.736352 3420 catalog_manager.cc:1347] Generated new cluster ID: efce8720b94c4dc4a63eee32854e86ed
Sep 26 05:52:15 dist-test-slave-7fr8 krb5kdc[3325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:52:15.736409 3420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:52:15.741062 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:15.727412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.24.255.62:45279)
0926 05:52:15.727655 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:15.727665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:15.727698 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:52:15.728820 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:15.728832 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:15.728906 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:15.729080 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:15.729865 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:15.731766 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:15.732967 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:15.733668 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:15.733803 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:15.737101 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:52:15.737106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:52:15.737107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:52:15.737120 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:52:15.739216 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:52:15.739725 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:52:15.739728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:52:15.739728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:52:15.739844 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:52:15.740059 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:52:15.740060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:52:15.740060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:52:15.740378 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:52:15.740552 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:15.740695 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 05:52:15.740842 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":143,"threads_started":1}
I20260926 05:52:15.744632 3420 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:52:15.745442 3420 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:52:15.752483 3420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1: Generated new TSK 0
I20260926 05:52:15.752698 3420 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:52:15.785640 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:45279
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:52:15.896364 3433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:52:15.896610 3433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:52:15.896646 3433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:52:15.897934 3433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:52:15.898017 3433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:52:15.898169 3433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:52:15.900154 3433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:52:15.900434 3433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:52:15.900663 3433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:52:15.905839 3439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:15.905839 3441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:15.906234 3433 server_base.cc:1034] running on GCE node
W20260926 05:52:15.906327 3438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:15.906579 3433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:52:15.906829 3433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:52:15.908006 3433 hybrid_clock.cc:648] HybridClock initialized: now 1790401935907991 us; error 29 us; skew 500 ppm
Sep 26 05:52:15 dist-test-slave-7fr8 krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401935, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:52:15.910905 3433 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:52:15.911407 3433 webserver.cc:460] Webserver started at http://127.24.255.1:39165/ using document root <none> and password file <none>
I20260926 05:52:15.911620 3433 fs_manager.cc:359] Metadata directory not provided
I20260926 05:52:15.911671 3433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:52:15.911798 3433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:52:15.912875 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "c03bab3004074c44b8311e03a6f0f1e5"
format_stamp: "Formatted at 2026-09-26 05:52:15 on dist-test-slave-7fr8"
server_key: "a4b2e8c677937c8e85f4bb8b395ca545"
server_key_iv: "e73f7314c994d495a96e4bc879244826"
server_key_version: "encryptionkey@0"
I20260926 05:52:15.913260 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "c03bab3004074c44b8311e03a6f0f1e5"
format_stamp: "Formatted at 2026-09-26 05:52:15 on dist-test-slave-7fr8"
server_key: "a4b2e8c677937c8e85f4bb8b395ca545"
server_key_iv: "e73f7314c994d495a96e4bc879244826"
server_key_version: "encryptionkey@0"
I20260926 05:52:15.914709 3433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:52:15.915524 3448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:15.915870 3433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:52:15.916067 3433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "c03bab3004074c44b8311e03a6f0f1e5"
format_stamp: "Formatted at 2026-09-26 05:52:15 on dist-test-slave-7fr8"
server_key: "a4b2e8c677937c8e85f4bb8b395ca545"
server_key_iv: "e73f7314c994d495a96e4bc879244826"
server_key_version: "encryptionkey@0"
I20260926 05:52:15.916177 3433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:52:15.928408 3433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:52:15.930757 3433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:52:15.930912 3433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:52:15.931156 3433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:52:15.931471 3433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:52:15.931514 3433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:15.931551 3433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:52:15.931576 3433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:15.937991 3433 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:32875
I20260926 05:52:15.938155 3561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:32875 every 8 connection(s)
I20260926 05:52:15.938421 3433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:52:15.941761 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 3433
I20260926 05:52:15.941876 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:52:15.942076 25596 external_mini_cluster.cc:1428] Setting key 8e98c2ec5db956a4afde91a113768f6f
Sep 26 05:52:15 dist-test-slave-7fr8 krb5kdc[3325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401935, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:52:15.950227 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:15.939220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:51607 (local address 127.24.255.62:45279)
0926 05:52:15.939270 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:15.939272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:15.939467 (+ 195us) server_negotiation.cc:407] Connection header received
0926 05:52:15.940203 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:15.940204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:15.940220 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:15.940271 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:15.941832 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:15.942618 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:15.943729 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:15.944012 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:15.944108 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:15.946508 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:52:15.946510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:52:15.946511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:52:15.946520 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:52:15.948318 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:52:15.948842 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:52:15.948843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:52:15.948844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:52:15.948914 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:52:15.949411 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:52:15.949412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:52:15.949412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:52:15.949582 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:52:15.949661 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:15.949981 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 05:52:15.950082 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:52:15.950331 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:15.939137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:45279 (local address 127.24.255.1:51607)
0926 05:52:15.939338 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:15.939384 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:15.939906 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:15.940478 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:15.940484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:15.941427 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:15.941674 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:15.941681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:15.942734 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:15.942735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:15.943631 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:15.943634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:15.944106 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:15.944108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:15.944245 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:15.944826 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:15.944836 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:15.946398 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:52:15.948501 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:52:15.948510 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:52:15.948513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:52:15.948761 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:52:15.949144 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:52:15.949145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:52:15.949145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:52:15.949293 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:52:15.949667 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:52:15.949677 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:52:15.949832 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 05:52:15.949936 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 05:52:15.950128 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
I20260926 05:52:15.951036 3562 heartbeater.cc:347] Connected to a master server at 127.24.255.62:45279
I20260926 05:52:15.951133 3562 heartbeater.cc:464] Registering TS with master...
I20260926 05:52:15.951327 3562 heartbeater.cc:511] Master 127.24.255.62:45279 requested a full tablet report, sending...
I20260926 05:52:15.951997 3370 ts_manager.cc:194] Registered new tserver with Master: c03bab3004074c44b8311e03a6f0f1e5 (127.24.255.1:32875)
I20260926 05:52:15.952653 3370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:51607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:52:15.969386 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:45279
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:52:16.053767 3567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:52:16.054005 3567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:52:16.054041 3567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:52:16.055384 3567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:52:16.055449 3567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:52:16.055480 3567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:52:16.057969 3567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:52:16.058320 3567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:52:16.058584 3567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:52:16.064006 3575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:16.064006 3573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:16.064250 3572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:16.064316 3567 server_base.cc:1034] running on GCE node
I20260926 05:52:16.064574 3567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:52:16.064845 3567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:52:16.066059 3567 hybrid_clock.cc:648] HybridClock initialized: now 1790401936066053 us; error 35 us; skew 500 ppm
Sep 26 05:52:16 dist-test-slave-7fr8 krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401936, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:52:16.069731 3567 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:52:16.070269 3567 webserver.cc:460] Webserver started at http://127.24.255.2:32809/ using document root <none> and password file <none>
I20260926 05:52:16.070513 3567 fs_manager.cc:359] Metadata directory not provided
I20260926 05:52:16.070564 3567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:52:16.070678 3567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:52:16.071738 3567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "a31a5656bbb1416191c72b49ab5d1946"
format_stamp: "Formatted at 2026-09-26 05:52:16 on dist-test-slave-7fr8"
server_key: "7297a1281d0f18f386ccad207b1e0d2c"
server_key_iv: "bed9a170ff36a1b14e2b87e3d3cdc98f"
server_key_version: "encryptionkey@0"
I20260926 05:52:16.072073 3567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "a31a5656bbb1416191c72b49ab5d1946"
format_stamp: "Formatted at 2026-09-26 05:52:16 on dist-test-slave-7fr8"
server_key: "7297a1281d0f18f386ccad207b1e0d2c"
server_key_iv: "bed9a170ff36a1b14e2b87e3d3cdc98f"
server_key_version: "encryptionkey@0"
I20260926 05:52:16.073433 3567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:52:16.074494 3582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:16.074705 3567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:52:16.074841 3567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "a31a5656bbb1416191c72b49ab5d1946"
format_stamp: "Formatted at 2026-09-26 05:52:16 on dist-test-slave-7fr8"
server_key: "7297a1281d0f18f386ccad207b1e0d2c"
server_key_iv: "bed9a170ff36a1b14e2b87e3d3cdc98f"
server_key_version: "encryptionkey@0"
I20260926 05:52:16.074922 3567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:52:16.093791 3567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:52:16.096045 3567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:52:16.096195 3567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:52:16.096426 3567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:52:16.096762 3567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:52:16.096797 3567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:16.096827 3567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:52:16.096851 3567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:16.102685 3567 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:41097
I20260926 05:52:16.102741 3695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:41097 every 8 connection(s)
I20260926 05:52:16.103209 3567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:52:16.105051 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 3567
I20260926 05:52:16.105136 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:52:16.105311 25596 external_mini_cluster.cc:1428] Setting key 58bd8b02372532d9ace6870a51342706
Sep 26 05:52:16 dist-test-slave-7fr8 krb5kdc[3325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401936, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
I20260926 05:52:16.113667 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.104042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:55379 (local address 127.24.255.62:45279)
0926 05:52:16.104187 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:16.104190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:16.104421 (+ 231us) server_negotiation.cc:407] Connection header received
0926 05:52:16.104895 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:16.104897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:16.104909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:16.104962 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:16.106259 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.106769 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.107838 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.108047 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.108104 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.110661 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:52:16.110663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:52:16.110663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:52:16.110672 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:52:16.112038 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:52:16.112570 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:52:16.112572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:52:16.112572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:52:16.112636 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:52:16.112938 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:52:16.112939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:52:16.112939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:52:16.113067 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:52:16.113122 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:16.113518 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 05:52:16.113594 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 05:52:16.113873 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.104013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:45279 (local address 127.24.255.2:55379)
0926 05:52:16.104247 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:16.104288 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:16.104799 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:16.105039 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:16.105043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:16.105805 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:16.106174 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.106181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.106866 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.106867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.107755 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.107758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.108118 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.108119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.108257 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.108774 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:16.108784 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:16.110531 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:52:16.112130 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:52:16.112139 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:52:16.112142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:52:16.112437 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:52:16.112711 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:52:16.112712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:52:16.112712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:52:16.112867 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:52:16.113131 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:52:16.113137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:52:16.113316 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 05:52:16.113446 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 05:52:16.113624 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
I20260926 05:52:16.114372 3696 heartbeater.cc:347] Connected to a master server at 127.24.255.62:45279
I20260926 05:52:16.114463 3696 heartbeater.cc:464] Registering TS with master...
I20260926 05:52:16.114653 3696 heartbeater.cc:511] Master 127.24.255.62:45279 requested a full tablet report, sending...
I20260926 05:52:16.115353 3370 ts_manager.cc:194] Registered new tserver with Master: a31a5656bbb1416191c72b49ab5d1946 (127.24.255.2:41097)
I20260926 05:52:16.115741 3370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:55379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:52:16.133963 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:45279
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:52:16.218271 3701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:52:16.218500 3701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:52:16.218525 3701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:52:16.219796 3701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:52:16.219882 3701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:52:16.219918 3701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:52:16.221516 3701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:52:16.221740 3701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:52:16.221930 3701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:52:16.227207 3701 server_base.cc:1034] running on GCE node
W20260926 05:52:16.227175 3707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:16.227178 3709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:52:16.227167 3706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:52:16.227661 3701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:52:16.227905 3701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:52:16.229096 3701 hybrid_clock.cc:648] HybridClock initialized: now 1790401936229080 us; error 28 us; skew 500 ppm
Sep 26 05:52:16 dist-test-slave-7fr8 krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401936, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:52:16.231797 3701 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:52:16.232332 3701 webserver.cc:460] Webserver started at http://127.24.255.3:40323/ using document root <none> and password file <none>
I20260926 05:52:16.232573 3701 fs_manager.cc:359] Metadata directory not provided
I20260926 05:52:16.232622 3701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:52:16.232779 3701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:52:16.234004 3701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "9a5519d51717458788b8ac230693eaef"
format_stamp: "Formatted at 2026-09-26 05:52:16 on dist-test-slave-7fr8"
server_key: "30b7e394bf664b40df249d8be76cb989"
server_key_iv: "4abe22598ca4180871389aa99712ec2c"
server_key_version: "encryptionkey@0"
I20260926 05:52:16.234369 3701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "9a5519d51717458788b8ac230693eaef"
format_stamp: "Formatted at 2026-09-26 05:52:16 on dist-test-slave-7fr8"
server_key: "30b7e394bf664b40df249d8be76cb989"
server_key_iv: "4abe22598ca4180871389aa99712ec2c"
server_key_version: "encryptionkey@0"
I20260926 05:52:16.235682 3701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:52:16.236439 3716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:16.236622 3701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:52:16.236703 3701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "9a5519d51717458788b8ac230693eaef"
format_stamp: "Formatted at 2026-09-26 05:52:16 on dist-test-slave-7fr8"
server_key: "30b7e394bf664b40df249d8be76cb989"
server_key_iv: "4abe22598ca4180871389aa99712ec2c"
server_key_version: "encryptionkey@0"
I20260926 05:52:16.236764 3701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:52:16.250581 3701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:52:16.253108 3701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:52:16.253229 3701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:52:16.253437 3701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:52:16.253728 3701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:52:16.253759 3701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:16.253782 3701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:52:16.253793 3701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:52:16.260545 3701 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:34119
I20260926 05:52:16.260603 3829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:34119 every 8 connection(s)
I20260926 05:52:16.260933 3701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:52:16 dist-test-slave-7fr8 krb5kdc[3325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401936, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:52:16.269277 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 3701
I20260926 05:52:16.269372 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:52:16.269547 25596 external_mini_cluster.cc:1428] Setting key 1a9dc9be954c616af50eb7a1cd4693a3
I20260926 05:52:16.272282 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.261772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:34007 (local address 127.24.255.62:45279)
0926 05:52:16.261830 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:16.261832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:16.261945 (+ 113us) server_negotiation.cc:407] Connection header received
0926 05:52:16.262886 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:16.262887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:16.262899 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:16.262956 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:16.263979 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.264544 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.265581 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.265780 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.265833 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.268295 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:52:16.268297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:52:16.268297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:52:16.268307 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:52:16.270139 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:52:16.270962 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:52:16.270963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:52:16.270964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:52:16.271067 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:52:16.271454 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:52:16.271455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:52:16.271456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:52:16.271621 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:52:16.271704 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:16.272056 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 05:52:16.272189 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:52:16.272357 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.261662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:45279 (local address 127.24.255.3:34007)
0926 05:52:16.261818 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:16.261861 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:16.262753 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:16.263029 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:16.263033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:16.263646 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:52:16.263857 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.263863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.264634 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.264635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.265472 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.265475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.265878 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.265879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.266008 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.266600 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:52:16.266609 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:52:16.268174 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:52:16.270370 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:52:16.270380 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:52:16.270383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:52:16.270826 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:52:16.271197 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:52:16.271198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:52:16.271198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:52:16.271342 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:52:16.271720 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:52:16.271727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:52:16.271878 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 05:52:16.271987 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 05:52:16.272142 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260926 05:52:16.273241 3830 heartbeater.cc:347] Connected to a master server at 127.24.255.62:45279
I20260926 05:52:16.273350 3830 heartbeater.cc:464] Registering TS with master...
I20260926 05:52:16.273516 3830 heartbeater.cc:511] Master 127.24.255.62:45279 requested a full tablet report, sending...
I20260926 05:52:16.273969 3370 ts_manager.cc:194] Registered new tserver with Master: 9a5519d51717458788b8ac230693eaef (127.24.255.3:34119)
I20260926 05:52:16.274473 3370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:34007
I20260926 05:52:16.282217 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:52:16.290640 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.283756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.24.255.62:45279)
0926 05:52:16.283937 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:16.283940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:16.284086 (+ 146us) server_negotiation.cc:407] Connection header received
0926 05:52:16.284169 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:16.284171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:16.284185 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:16.284248 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:52:16.284894 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.285446 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.286268 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.286464 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.286524 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.288127 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:52:16.288129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:52:16.288130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:52:16.288142 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:52:16.289400 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:52:16.289751 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:52:16.289752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:52:16.289753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:52:16.289816 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:52:16.289984 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:52:16.289985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:52:16.289985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:52:16.290213 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:52:16.290309 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:16.290433 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 05:52:16.290522 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260926 05:52:16.292768 3370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:52:16.294426 3370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:52:16.301232 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.296805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:34119 (local address 127.0.0.1:56778)
0926 05:52:16.297270 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:16.297309 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:16.297399 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:16.297673 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:16.297676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:16.297678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:52:16.297760 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.297762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.298714 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.298715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.299989 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.299993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.300897 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.300898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.300979 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.301008 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 05:52:16.301079 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 05:52:16.301142 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":6,"thread_start_us":94,"threads_started":1}
I20260926 05:52:16.301234 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.296902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.24.255.3:34119)
0926 05:52:16.297189 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:16.297192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:16.297369 (+ 177us) server_negotiation.cc:407] Connection header received
0926 05:52:16.297490 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:16.297492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:16.297509 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:16.297559 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:52:16.297856 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.298617 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.300088 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.300808 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.300890 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.300949 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:16.301105 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 05:52:16.301148 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":86,"threads_started":1}
I20260926 05:52:16.301357 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.296962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.24.255.1:32875)
0926 05:52:16.297124 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:16.297128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:16.297367 (+ 239us) server_negotiation.cc:407] Connection header received
0926 05:52:16.297494 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:16.297497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:16.297511 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:16.297575 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:52:16.297854 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.298996 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.300361 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.300955 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.300999 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.301050 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:16.301242 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 05:52:16.301285 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260926 05:52:16.301545 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.296801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:32875 (local address 127.0.0.1:37918)
0926 05:52:16.297256 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:16.297297 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:16.297374 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:16.297676 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:16.297677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:16.297680 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:52:16.297772 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.297774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.299114 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.299115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.300266 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.300269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.301106 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.301106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.301159 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.301162 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:52:16.301428 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 05:52:16.301475 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":4,"thread_start_us":51,"threads_started":1}
I20260926 05:52:16.301795 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.296995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:41097 (local address 127.0.0.1:59264)
0926 05:52:16.297346 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:16.297354 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:16.297410 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:16.297733 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:16.297733 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:16.297735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:52:16.297771 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.297774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.299122 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.299122 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.300448 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.300450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.301499 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.301500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.301561 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.301566 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:52:16.301674 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 05:52:16.301728 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":53,"thread_start_us":63,"threads_started":1}
I20260926 05:52:16.302040 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.297108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.24.255.2:41097)
0926 05:52:16.297253 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:16.297258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:16.297406 (+ 148us) server_negotiation.cc:407] Connection header received
0926 05:52:16.297476 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:16.297479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:16.297494 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:16.297668 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:52:16.297848 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.299025 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.300527 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.301173 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.301235 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.301283 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:16.301928 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 05:52:16.301971 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260926 05:52:16.302062 3764 tablet_service.cc:1467] Processing CreateTablet for tablet 8495556b95da453b8f8f5d82e6a1de8a (DEFAULT_TABLE table=test-table [id=2c3dc16255774b71afb06e4589722ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:52:16.302207 3496 tablet_service.cc:1467] Processing CreateTablet for tablet 8495556b95da453b8f8f5d82e6a1de8a (DEFAULT_TABLE table=test-table [id=2c3dc16255774b71afb06e4589722ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:52:16.302495 3764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8495556b95da453b8f8f5d82e6a1de8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:52:16.302538 3496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8495556b95da453b8f8f5d82e6a1de8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:52:16.302793 3630 tablet_service.cc:1467] Processing CreateTablet for tablet 8495556b95da453b8f8f5d82e6a1de8a (DEFAULT_TABLE table=test-table [id=2c3dc16255774b71afb06e4589722ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:52:16.303085 3630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8495556b95da453b8f8f5d82e6a1de8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:52:16.305054 3847 tablet_bootstrap.cc:492] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef: Bootstrap starting.
I20260926 05:52:16.305775 3847 tablet_bootstrap.cc:654] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef: Neither blocks nor log segments found. Creating new log.
I20260926 05:52:16.305830 3848 tablet_bootstrap.cc:492] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5: Bootstrap starting.
I20260926 05:52:16.306376 3847 log.cc:824] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef: Log is configured to *not* fsync() on all Append() calls
I20260926 05:52:16.306367 3849 tablet_bootstrap.cc:492] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946: Bootstrap starting.
I20260926 05:52:16.306483 3848 tablet_bootstrap.cc:654] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5: Neither blocks nor log segments found. Creating new log.
I20260926 05:52:16.307008 3848 log.cc:824] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5: Log is configured to *not* fsync() on all Append() calls
I20260926 05:52:16.307039 3849 tablet_bootstrap.cc:654] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946: Neither blocks nor log segments found. Creating new log.
I20260926 05:52:16.307579 3849 log.cc:824] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946: Log is configured to *not* fsync() on all Append() calls
I20260926 05:52:16.307833 3848 tablet_bootstrap.cc:492] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5: No bootstrap required, opened a new log
I20260926 05:52:16.307845 3847 tablet_bootstrap.cc:492] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef: No bootstrap required, opened a new log
I20260926 05:52:16.307899 3848 ts_tablet_manager.cc:1397] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:52:16.307945 3847 ts_tablet_manager.cc:1397] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:52:16.308322 3849 tablet_bootstrap.cc:492] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946: No bootstrap required, opened a new log
I20260926 05:52:16.308391 3849 ts_tablet_manager.cc:1397] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:52:16.309670 3847 raft_consensus.cc:348] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5519d51717458788b8ac230693eaef" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 34119 } } peers { permanent_uuid: "c03bab3004074c44b8311e03a6f0f1e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 32875 } } peers { permanent_uuid: "a31a5656bbb1416191c72b49ab5d1946" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41097 } }
I20260926 05:52:16.309813 3847 raft_consensus.cc:374] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:52:16.309844 3847 raft_consensus.cc:729] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5519d51717458788b8ac230693eaef, State: Initialized, Role: FOLLOWER
I20260926 05:52:16.309864 3848 raft_consensus.cc:348] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5519d51717458788b8ac230693eaef" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 34119 } } peers { permanent_uuid: "c03bab3004074c44b8311e03a6f0f1e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 32875 } } peers { permanent_uuid: "a31a5656bbb1416191c72b49ab5d1946" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41097 } }
I20260926 05:52:16.310003 3847 consensus_queue.cc:260] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5519d51717458788b8ac230693eaef" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 34119 } } peers { permanent_uuid: "c03bab3004074c44b8311e03a6f0f1e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 32875 } } peers { permanent_uuid: "a31a5656bbb1416191c72b49ab5d1946" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41097 } }
I20260926 05:52:16.310056 3848 raft_consensus.cc:374] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:52:16.310122 3848 raft_consensus.cc:729] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03bab3004074c44b8311e03a6f0f1e5, State: Initialized, Role: FOLLOWER
I20260926 05:52:16.310293 3847 ts_tablet_manager.cc:1428] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:52:16.310267 3848 consensus_queue.cc:260] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5519d51717458788b8ac230693eaef" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 34119 } } peers { permanent_uuid: "c03bab3004074c44b8311e03a6f0f1e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 32875 } } peers { permanent_uuid: "a31a5656bbb1416191c72b49ab5d1946" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41097 } }
I20260926 05:52:16.310403 3830 heartbeater.cc:503] Master 127.24.255.62:45279 was elected leader, sending a full tablet report...
I20260926 05:52:16.310307 3849 raft_consensus.cc:348] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5519d51717458788b8ac230693eaef" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 34119 } } peers { permanent_uuid: "c03bab3004074c44b8311e03a6f0f1e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 32875 } } peers { permanent_uuid: "a31a5656bbb1416191c72b49ab5d1946" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41097 } }
I20260926 05:52:16.310442 3849 raft_consensus.cc:374] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:52:16.310474 3849 raft_consensus.cc:729] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a31a5656bbb1416191c72b49ab5d1946, State: Initialized, Role: FOLLOWER
I20260926 05:52:16.310526 3848 ts_tablet_manager.cc:1428] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:52:16.310631 3562 heartbeater.cc:503] Master 127.24.255.62:45279 was elected leader, sending a full tablet report...
I20260926 05:52:16.310621 3849 consensus_queue.cc:260] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5519d51717458788b8ac230693eaef" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 34119 } } peers { permanent_uuid: "c03bab3004074c44b8311e03a6f0f1e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 32875 } } peers { permanent_uuid: "a31a5656bbb1416191c72b49ab5d1946" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41097 } }
I20260926 05:52:16.310880 3696 heartbeater.cc:503] Master 127.24.255.62:45279 was elected leader, sending a full tablet report...
I20260926 05:52:16.310875 3849 ts_tablet_manager.cc:1428] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260926 05:52:16.353942 3697 tablet.cc:2369] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:52:16.439198 3563 tablet.cc:2369] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:52:16.511543 3831 tablet.cc:2369] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:52:16.727628 3853 raft_consensus.cc:482] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:52:16.727783 3853 raft_consensus.cc:504] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5519d51717458788b8ac230693eaef" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 34119 } } peers { permanent_uuid: "c03bab3004074c44b8311e03a6f0f1e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 32875 } } peers { permanent_uuid: "a31a5656bbb1416191c72b49ab5d1946" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41097 } }
I20260926 05:52:16.728173 3853 leader_election.cc:288] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c03bab3004074c44b8311e03a6f0f1e5 (127.24.255.1:32875), a31a5656bbb1416191c72b49ab5d1946 (127.24.255.2:41097)
I20260926 05:52:16.731477 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.728514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:33089 (local address 127.24.255.1:32875)
0926 05:52:16.728566 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:16.728568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:16.728580 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:52:16.728666 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:16.728668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:16.728681 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:16.728740 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:52:16.729016 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.729553 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.730727 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.731127 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.731182 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.731190 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:16.731366 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 05:52:16.731402 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:52:16.731510 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.728408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:32875 (local address 127.24.255.3:33089)
0926 05:52:16.728469 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:16.728482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:16.728598 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:16.728750 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:16.728750 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:16.728752 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:52:16.728902 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.728906 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.729653 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.729654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.730620 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.730623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.731210 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.731211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.731297 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.731302 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:52:16.731357 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 05:52:16.731402 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:52:16.731679 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.728537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:41097 (local address 127.24.255.3:60821)
0926 05:52:16.728722 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:16.728731 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:52:16.728777 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:52:16.729014 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:52:16.729015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:52:16.729017 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:52:16.729085 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.729087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.729885 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.729886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.730725 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:52:16.730727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.731437 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.731437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:52:16.731508 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.731512 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:52:16.731568 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 05:52:16.731607 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":96,"threads_started":1}
I20260926 05:52:16.731823 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.728645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:60821 (local address 127.24.255.2:41097)
0926 05:52:16.728690 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:16.728691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:16.728788 (+ 97us) server_negotiation.cc:407] Connection header received
0926 05:52:16.728889 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:16.728890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:16.728903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:16.728985 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:52:16.729187 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.729772 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.730852 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.731290 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.731367 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.731375 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:16.731717 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 05:52:16.731762 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:52:16.731973 3516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8495556b95da453b8f8f5d82e6a1de8a" candidate_uuid: "9a5519d51717458788b8ac230693eaef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03bab3004074c44b8311e03a6f0f1e5" is_pre_election: true
I20260926 05:52:16.732095 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8495556b95da453b8f8f5d82e6a1de8a" candidate_uuid: "9a5519d51717458788b8ac230693eaef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a31a5656bbb1416191c72b49ab5d1946" is_pre_election: true
I20260926 05:52:16.732153 3516 raft_consensus.cc:2454] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a5519d51717458788b8ac230693eaef in term 0.
I20260926 05:52:16.732230 3650 raft_consensus.cc:2454] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a5519d51717458788b8ac230693eaef in term 0.
I20260926 05:52:16.732398 3718 leader_election.cc:302] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a5519d51717458788b8ac230693eaef, c03bab3004074c44b8311e03a6f0f1e5; no voters:
I20260926 05:52:16.732517 3853 raft_consensus.cc:2781] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:52:16.732616 3853 raft_consensus.cc:482] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:52:16.732650 3853 raft_consensus.cc:3037] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:52:16.733392 3853 raft_consensus.cc:504] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5519d51717458788b8ac230693eaef" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 34119 } } peers { permanent_uuid: "c03bab3004074c44b8311e03a6f0f1e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 32875 } } peers { permanent_uuid: "a31a5656bbb1416191c72b49ab5d1946" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41097 } }
I20260926 05:52:16.733518 3853 leader_election.cc:288] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [CANDIDATE]: Term 1 election: Requested vote from peers c03bab3004074c44b8311e03a6f0f1e5 (127.24.255.1:32875), a31a5656bbb1416191c72b49ab5d1946 (127.24.255.2:41097)
I20260926 05:52:16.733721 3516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8495556b95da453b8f8f5d82e6a1de8a" candidate_uuid: "9a5519d51717458788b8ac230693eaef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03bab3004074c44b8311e03a6f0f1e5"
I20260926 05:52:16.733753 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8495556b95da453b8f8f5d82e6a1de8a" candidate_uuid: "9a5519d51717458788b8ac230693eaef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a31a5656bbb1416191c72b49ab5d1946"
I20260926 05:52:16.733806 3516 raft_consensus.cc:3037] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:52:16.733830 3650 raft_consensus.cc:3037] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:52:16.734602 3650 raft_consensus.cc:2454] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a5519d51717458788b8ac230693eaef in term 1.
I20260926 05:52:16.734627 3516 raft_consensus.cc:2454] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a5519d51717458788b8ac230693eaef in term 1.
I20260926 05:52:16.734771 3718 leader_election.cc:302] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a5519d51717458788b8ac230693eaef, a31a5656bbb1416191c72b49ab5d1946; no voters:
I20260926 05:52:16.735031 3853 raft_consensus.cc:2781] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:52:16.735179 3853 raft_consensus.cc:686] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [term 1 LEADER]: Becoming Leader. State: Replica: 9a5519d51717458788b8ac230693eaef, State: Running, Role: LEADER
I20260926 05:52:16.735376 3853 consensus_queue.cc:237] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5519d51717458788b8ac230693eaef" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 34119 } } peers { permanent_uuid: "c03bab3004074c44b8311e03a6f0f1e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 32875 } } peers { permanent_uuid: "a31a5656bbb1416191c72b49ab5d1946" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41097 } }
I20260926 05:52:16.736339 3368 catalog_manager.cc:5495] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a5519d51717458788b8ac230693eaef (127.24.255.3). New cstate: current_term: 1 leader_uuid: "9a5519d51717458788b8ac230693eaef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5519d51717458788b8ac230693eaef" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c03bab3004074c44b8311e03a6f0f1e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 32875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a31a5656bbb1416191c72b49ab5d1946" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41097 } health_report { overall_health: UNKNOWN } } }
I20260926 05:52:16.743247 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.740904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.24.255.3:34119)
0926 05:52:16.740970 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:52:16.740972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:52:16.740980 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:52:16.741015 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:52:16.741016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:52:16.741031 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:52:16.741089 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:52:16.741207 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.741741 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.742554 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:52:16.742743 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:52:16.742799 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:52:16.742930 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:52:16.743031 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:52:16.743103 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:52:16.743129 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 05:52:16.743153 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 05:52:16.745034 3516 raft_consensus.cc:1261] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5 [term 1 FOLLOWER]: Refusing update from remote peer 9a5519d51717458788b8ac230693eaef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:52:16.745031 3650 raft_consensus.cc:1261] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946 [term 1 FOLLOWER]: Refusing update from remote peer 9a5519d51717458788b8ac230693eaef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:52:16.745385 3853 consensus_queue.cc:1035] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03bab3004074c44b8311e03a6f0f1e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 32875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:52:16.745579 3857 consensus_queue.cc:1035] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [LEADER]: Connected to new peer: Peer: permanent_uuid: "a31a5656bbb1416191c72b49ab5d1946" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:52:16.748742 3862 mvcc.cc:205] Tried to move back new op lower bound from 7333486332904734720 to 7333486332868767744. Current Snapshot: MvccSnapshot[applied={T|T < 7333486332904734720}]
I20260926 05:52:16.751142 3369 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:52:16.751271 3369 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:52:16.752126 3369 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01166a55c66642fc845b5c9dd380f7e1: Sending DeleteTablet for 3 replicas of tablet 8495556b95da453b8f8f5d82e6a1de8a
I20260926 05:52:16.752439 3764 tablet_service.cc:1514] Processing DeleteTablet for tablet 8495556b95da453b8f8f5d82e6a1de8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:52:16 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:56778
I20260926 05:52:16.752525 3630 tablet_service.cc:1514] Processing DeleteTablet for tablet 8495556b95da453b8f8f5d82e6a1de8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:52:16 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:59264
I20260926 05:52:16.752552 3496 tablet_service.cc:1514] Processing DeleteTablet for tablet 8495556b95da453b8f8f5d82e6a1de8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:52:16 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:37918
I20260926 05:52:16.752727 3871 tablet_replica.cc:330] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946: stopping tablet replica
I20260926 05:52:16.752827 3871 raft_consensus.cc:2229] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:52:16.752853 3870 tablet_replica.cc:330] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef: stopping tablet replica
I20260926 05:52:16.752945 3871 raft_consensus.cc:2258] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:52:16.753099 3872 tablet_replica.cc:330] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5: stopping tablet replica
I20260926 05:52:16.753199 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 3433
I20260926 05:52:16.753206 3872 raft_consensus.cc:2229] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:52:16.753311 3872 raft_consensus.cc:2258] T 8495556b95da453b8f8f5d82e6a1de8a P c03bab3004074c44b8311e03a6f0f1e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:52:16.753701 3871 ts_tablet_manager.cc:1905] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:52:16.754894 3870 raft_consensus.cc:2229] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:52:16.755079 3870 raft_consensus.cc:2258] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:52:16.755600 3871 ts_tablet_manager.cc:1918] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:52:16.755616 3870 ts_tablet_manager.cc:1905] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:52:16.755684 3871 log.cc:1196] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/8495556b95da453b8f8f5d82e6a1de8a
I20260926 05:52:16.755949 3871 ts_tablet_manager.cc:1939] T 8495556b95da453b8f8f5d82e6a1de8a P a31a5656bbb1416191c72b49ab5d1946: Deleting consensus metadata
I20260926 05:52:16.756574 3355 catalog_manager.cc:4841] TS a31a5656bbb1416191c72b49ab5d1946 (127.24.255.2:41097): tablet 8495556b95da453b8f8f5d82e6a1de8a (table test-table [id=2c3dc16255774b71afb06e4589722ab3]) successfully deleted
I20260926 05:52:16.757416 3870 ts_tablet_manager.cc:1918] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:52:16.757531 3870 log.cc:1196] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/8495556b95da453b8f8f5d82e6a1de8a
I20260926 05:52:16.757790 3870 ts_tablet_manager.cc:1939] T 8495556b95da453b8f8f5d82e6a1de8a P 9a5519d51717458788b8ac230693eaef: Deleting consensus metadata
I20260926 05:52:16.758376 3356 catalog_manager.cc:4841] TS 9a5519d51717458788b8ac230693eaef (127.24.255.3:34119): tablet 8495556b95da453b8f8f5d82e6a1de8a (table test-table [id=2c3dc16255774b71afb06e4589722ab3]) successfully deleted
W20260926 05:52:16.759728 3355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:32875 (error 108)
I20260926 05:52:16.759907 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 3567
I20260926 05:52:16.760812 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:52:16.760676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:32875 (local address 127.0.0.1:37934)
0926 05:52:16.760732 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:52:16.760776 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 05:52:16.760984 3355 catalog_manager.cc:4584] TS c03bab3004074c44b8311e03a6f0f1e5 (127.24.255.1:32875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8495556b95da453b8f8f5d82e6a1de8a: Network error: Client connection negotiation failed: client connection to 127.24.255.1:32875: connect: Connection refused (error 111)
I20260926 05:52:16.765336 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 3701
I20260926 05:52:16.769568 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 3338
2026-09-26T05:52:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2274 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:52:16 dist-test-slave-7fr8 krb5kdc[3876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:52:16 dist-test-slave-7fr8 krb5kdc[3876](info): set up 2 sockets
Sep 26 05:52:16 dist-test-slave-7fr8 krb5kdc[3876](info): commencing operation
krb5kdc: starting...
W20260926 05:52:17.743139 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:52:17 dist-test-slave-7fr8 krb5kdc[3876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:52:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.255.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.255.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/rangeradmin_127.24.255.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/rangeradmin_127.24.255.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.255.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.255.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/rangerlookup_127.24.255.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/rangerlookup_127.24.255.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.255.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.255.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/HTTP_127.24.255.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/HTTP_127.24.255.20@KRBTEST.COM.keytab.
I20260926 05:52:17.909116 25596 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGUnhcE/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/postgres -l logfile start
2026-09-26 05:52:20.828 UTC [3906] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 05:52:20.828 UTC [3906] LOG: listening on IPv4 address "127.24.255.20", port 56019
2026-09-26 05:52:20.833 UTC [3906] LOG: listening on Unix socket "/tmp/.s.PGSQL.56019"
2026-09-26 05:52:20.862 UTC [3909] LOG: database system was shut down at 2026-09-26 05:52:18 UTC
2026-09-26 05:52:20.866 UTC [3906] LOG: database system is ready to accept connections
I20260926 05:52:21.747383 25596 mini_postgres.cc:96] Postgres bound to 56019
2026-09-26 05:52:21.753 UTC [3917] FATAL: database "slave" does not exist
127.24.255.20:56019 - accepting connections
I20260926 05:52:21.753860 25596 mini_ranger.cc:161] Starting Ranger...
I20260926 05:52:21.768953 25596 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 05:52:22.319144 25596 mini_ranger.cc:87] Created ranger Postgres database
I20260926 05:52:22.319234 25596 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-admin
2026-09-26 05:52:22,609 [I] DB FLAVOR :POSTGRES
2026-09-26 05:52:22,609 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 05:52:22,609 [I] Checking connection..
2026-09-26 05:52:22,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 05:52:22,880 [I] Checking connection passed.
2026-09-26 05:52:22,880 [I] --------- Verifying version history table ---------
2026-09-26 05:52:22,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:52:23,153 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 05:52:23,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:52:23,420 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 05:52:23,420 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 05:52:23,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 05:52:23.663 UTC [3981] WARNING: there is no transaction in progress
2026-09-26 05:52:23,669 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 05:52:23,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:52:23,926 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 05:52:23,926 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 05:52:23,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 05:52:24,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 05:52:24,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7fr8.c.gcp-upstream.internal','N') ;"
2026-09-26 05:52:24,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 05:52:24,982 [I] Table x_portal_user does not exist in database ranger
2026-09-26 05:52:24,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 05:52:25,251 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 05:52:25,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 05:52:25,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 05:52:25,794 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 05:52:25,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 05:52:26.899 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:26.918 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:26.938 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:26.957 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:26.978 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:27.001 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:27.067 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:27.076 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:27.093 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:27.105 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:27.119 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:27.137 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:27.150 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:27.158 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:27.167 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:27.175 UTC [4089] WARNING: there is no transaction in progress
2026-09-26 05:52:27,990 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 05:52:27,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 05:52:28,260 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 05:52:28,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 05:52:28,544 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 05:52:28,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 05:52:28,817 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 05:52:28,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 05:52:29,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 05:52:29,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7fr8.c.gcp-upstream.internal';"
2026-09-26 05:52:29,603 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 05:52:29,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 05:52:29,878 [I] DB_PATCHES have already been applied
I20260926 05:52:29.884361 25596 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGUnhcE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 05:52:29.884430 25596 mini_ranger.cc:193] Using host: 127.24.255.20
I20260926 05:52:29.887512 25596 mini_ranger.cc:227] Ranger admin URL: http://127.24.255.20:52425
Sep 26, 2026 5:52:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 5:52:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 5:52:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 5:52:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 5:52:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 5:52:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 5:52:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 05:52:31.964537 25596 mini_ranger.cc:160] Time spent starting Ranger: real 10.211s user 0.005s sys 0.005s
Sep 26, 2026 5:52:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 5:52:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 5:52:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 5:52:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 5:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 5:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 5:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 5:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 5:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 5:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 5:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 5:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 5:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 5:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 5:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 5:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 5:52:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 5:52:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 05:52:53.456303 25596 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.255.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.255.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/rangerkms_127.24.255.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/rangerkms_127.24.255.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.255.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.255.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.255.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/HTTP_127.24.255.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.255.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/HTTP_127.24.255.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 05:52:53 dist-test-slave-7fr8 krb5kdc[3876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401973, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 05:52:53.545051 25596 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 05:52:53.555583 25596 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 05:52:54.700134 25596 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 05:52:54.700210 25596 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-kms
I20260926 05:52:54.700227 25596 mini_ranger_kms.cc:227] Using postgres at 127.24.255.20:56019
2026-09-26 05:52:54,881 [I] DB FLAVOR :POSTGRES
2026-09-26 05:52:54,881 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 05:52:54,881 [I] Checking connection
2026-09-26 05:52:54,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 05:52:55,145 [I] connection success
2026-09-26 05:52:55,145 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 05:52:55,145 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 05:52:55,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 05:52:55,389 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 05:52:55,389 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 05:52:55,390 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 05:52:55,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:56019/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 05:52:55,692 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 05:52:56.204677 25596 mini_ranger_kms.cc:310] Created kms service
I20260926 05:52:56.450085 25596 mini_ranger_kms.cc:326] Created kudu user
I20260926 05:52:56.498507 25596 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 05:52:56 dist-test-slave-7fr8 krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401973, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:52:56.969027 25596 mini_ranger_kms.cc:384] Added ranger policy
I20260926 05:52:56.969321 25596 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 05:52:56.969352 25596 mini_ranger_kms.cc:240] Using host: 127.24.255.20
I20260926 05:52:56.972882 25596 mini_ranger_kms.cc:276] Ranger KMS PID: 4271
I20260926 05:52:56.972986 25596 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.255.20:59827
Sep 26, 2026 5:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 5:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 5:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 5:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 5:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 5:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 05:52:59.069124 25596 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.524s user 0.007s sys 0.157s
I20260926 05:52:59.069228 25596 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 05:52:59.069308 25596 mini_ranger_kms.cc:401] 127.24.255.20:59827/kms/v1/keys
Sep 26 05:52:59 dist-test-slave-7fr8 krb5kdc[3876](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790401979, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.255.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 05:52:59 dist-test-slave-7fr8 krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401979, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.255.20@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
Sep 26 05:53:00 dist-test-slave-7fr8 krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401979, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.255.20@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
Sep 26 05:53:02 dist-test-slave-7fr8 krb5kdc[3876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:53:02.754004 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:41143
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:41143
--ranger_config_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.255.20:59827/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:53:02.845155 4321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:02.845685 4321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:02.845731 4321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:02.848389 4321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:53:02.848488 4321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:02.848513 4321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:02.848532 4321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:53:02.848547 4321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:53:02.850850 4321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.255.20:59827/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:41143
--ranger_config_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:41143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:02.851148 4321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:02.851404 4321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:53:02.856479 4327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:02.856581 4329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:02.856482 4326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:02.857322 4321 server_base.cc:1034] running on GCE node
I20260926 05:53:02.857656 4321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:02.858088 4321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:02.859350 4321 hybrid_clock.cc:648] HybridClock initialized: now 1790401982859330 us; error 34 us; skew 500 ppm
Sep 26 05:53:02 dist-test-slave-7fr8 krb5kdc[3876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401982, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:02.862895 4321 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:53:02.863397 4321 webserver.cc:460] Webserver started at http://127.24.255.62:42861/ using document root <none> and password file <none>
I20260926 05:53:02.863633 4321 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:02.863677 4321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:53:02.863804 4321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:53:02 dist-test-slave-7fr8 krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401982, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:02.926178 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "757168f6ab564365b3ed9d12fa651fdc"
format_stamp: "Formatted at 2026-09-26 05:53:02 on dist-test-slave-7fr8"
server_key: "6018e7a5a7a857678c0e66641d640dea"
server_key_iv: "defdbab2bec2aaace6b6232a259772ec"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:02.926594 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "757168f6ab564365b3ed9d12fa651fdc"
format_stamp: "Formatted at 2026-09-26 05:53:02 on dist-test-slave-7fr8"
server_key: "6018e7a5a7a857678c0e66641d640dea"
server_key_iv: "defdbab2bec2aaace6b6232a259772ec"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:02.927940 4321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:53:02.945087 4336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:02.945298 4321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:53:02.945401 4321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "757168f6ab564365b3ed9d12fa651fdc"
format_stamp: "Formatted at 2026-09-26 05:53:02 on dist-test-slave-7fr8"
server_key: "6018e7a5a7a857678c0e66641d640dea"
server_key_iv: "defdbab2bec2aaace6b6232a259772ec"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:02.945499 4321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:02.976651 4321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:02.979173 4321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:02.979384 4321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:02.986083 4321 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:41143
I20260926 05:53:02.986675 4321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:53:02.986129 4388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:41143 every 8 connection(s)
I20260926 05:53:02.987524 4389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:53:02.990983 4389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc: Bootstrap starting.
I20260926 05:53:02.991628 4389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc: Neither blocks nor log segments found. Creating new log.
I20260926 05:53:02.991922 4389 log.cc:824] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc: Log is configured to *not* fsync() on all Append() calls
I20260926 05:53:02.993067 4389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc: No bootstrap required, opened a new log
I20260926 05:53:02.993618 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 4321
I20260926 05:53:02.993716 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:53:02.993921 25596 external_mini_cluster.cc:1428] Setting key 4a32cd8f8d827d4da6244c4e374e27c0
I20260926 05:53:02.995664 4389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757168f6ab564365b3ed9d12fa651fdc" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 41143 } }
I20260926 05:53:02.995818 4389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:53:02.995850 4389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757168f6ab564365b3ed9d12fa651fdc, State: Initialized, Role: FOLLOWER
I20260926 05:53:02.996019 4389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757168f6ab564365b3ed9d12fa651fdc" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 41143 } }
I20260926 05:53:02.996096 4389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:53:02.996129 4389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:53:02.996188 4389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:53:02.996799 4389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757168f6ab564365b3ed9d12fa651fdc" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 41143 } }
I20260926 05:53:02.996906 4389 leader_election.cc:302] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 757168f6ab564365b3ed9d12fa651fdc; no voters:
I20260926 05:53:02.997110 4389 leader_election.cc:288] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:53:02.997152 4394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:53:02.997347 4394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [term 1 LEADER]: Becoming Leader. State: Replica: 757168f6ab564365b3ed9d12fa651fdc, State: Running, Role: LEADER
I20260926 05:53:02.997460 4389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:53:02.997476 4394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757168f6ab564365b3ed9d12fa651fdc" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 41143 } }
I20260926 05:53:02.997866 4394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "757168f6ab564365b3ed9d12fa651fdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757168f6ab564365b3ed9d12fa651fdc" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 41143 } } }
I20260926 05:53:02.998144 4394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [sys.catalog]: This master's current role is: LEADER
I20260926 05:53:02.998320 4394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 757168f6ab564365b3ed9d12fa651fdc. Latest consensus state: current_term: 1 leader_uuid: "757168f6ab564365b3ed9d12fa651fdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757168f6ab564365b3ed9d12fa651fdc" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 41143 } } }
I20260926 05:53:02.998435 4394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc [sys.catalog]: This master's current role is: LEADER
I20260926 05:53:02.998901 4389 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 26 05:53:02 dist-test-slave-7fr8 krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:53:03.006319 4403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:53:03.006942 4403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:53:03.010952 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:02.994515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.24.255.62:41143)
0926 05:53:02.994779 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:02.994790 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:02.994825 (+ 35us) server_negotiation.cc:407] Connection header received
0926 05:53:02.995668 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:02.995679 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:02.995756 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:02.996686 (+ 930us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:02.996807 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:02.998317 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:02.999334 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:03.002308 (+ 2974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:03.002465 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:03.005406 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:03.005412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:03.005413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:03.005428 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:03.009034 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:03.009533 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:03.009538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:03.009538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:03.009653 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:03.009878 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:03.009879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:03.009879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:03.010234 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:03.010382 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:03.010565 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 05:53:03.010705 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":88,"threads_started":1}
I20260926 05:53:03.015538 4403 catalog_manager.cc:1347] Generated new cluster ID: 3ea0314db60b4ff0945955652be94d51
I20260926 05:53:03.015621 4403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:53:03.049609 4403 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:53:03.050498 4403 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:53:03.066833 4403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc: Generated new TSK 0
I20260926 05:53:03.067173 4403 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:53:03.942725 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.255.20:59827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:41143
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:53:04.074465 4434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:04.074940 4434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:04.075021 4434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:04.076421 4434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:04.076474 4434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:04.076517 4434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:53:04.078442 4434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.255.20:59827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:04.078722 4434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:04.079083 4434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:53:04.085644 4442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:04.085625 4440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:04.085628 4439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:04.086472 4434 server_base.cc:1034] running on GCE node
I20260926 05:53:04.086733 4434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:04.087055 4434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:04.088238 4434 hybrid_clock.cc:648] HybridClock initialized: now 1790401984088220 us; error 33 us; skew 500 ppm
Sep 26 05:53:04 dist-test-slave-7fr8 krb5kdc[3876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401984, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:04.096616 4434 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:53:04.097222 4434 webserver.cc:460] Webserver started at http://127.24.255.1:39469/ using document root <none> and password file <none>
I20260926 05:53:04.097513 4434 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:04.097582 4434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:53:04.097765 4434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:53:04 dist-test-slave-7fr8 krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401984, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:04.113020 4434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "836f18151e3448148ff9ba2d06f70ae5"
format_stamp: "Formatted at 2026-09-26 05:53:04 on dist-test-slave-7fr8"
server_key: "3cdc47a4e83b7176c9b04041a799c431"
server_key_iv: "904e70c312f7a8f8a683cd1a6c76868c"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:04.113571 4434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "836f18151e3448148ff9ba2d06f70ae5"
format_stamp: "Formatted at 2026-09-26 05:53:04 on dist-test-slave-7fr8"
server_key: "3cdc47a4e83b7176c9b04041a799c431"
server_key_iv: "904e70c312f7a8f8a683cd1a6c76868c"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:04.115497 4434 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260926 05:53:04.131954 4451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:04.132207 4434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 05:53:04.132303 4434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "836f18151e3448148ff9ba2d06f70ae5"
format_stamp: "Formatted at 2026-09-26 05:53:04 on dist-test-slave-7fr8"
server_key: "3cdc47a4e83b7176c9b04041a799c431"
server_key_iv: "904e70c312f7a8f8a683cd1a6c76868c"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:04.132413 4434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:04.180552 4434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:04.183147 4434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:04.183308 4434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:04.183538 4434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:53:04.183940 4434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:53:04.183990 4434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:04.184032 4434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:53:04.184055 4434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:04.192655 4434 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:37607
I20260926 05:53:04.193228 4434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:53:04.202448 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 4434
I20260926 05:53:04.202569 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:53:04.202765 25596 external_mini_cluster.cc:1428] Setting key 16f66d8ec2115b5ce39a6a6b8db3ee1b
I20260926 05:53:04.203610 4564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:37607 every 8 connection(s)
Sep 26 05:53:04 dist-test-slave-7fr8 krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401984, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:53:04.216830 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.203132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:41143 (local address 127.24.255.1:53227)
0926 05:53:04.204430 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:04.204502 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:04.205119 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:04.205564 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:04.205570 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:04.205839 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:04.206172 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.206178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.207363 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.207364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.208750 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.208754 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.209312 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.209313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.209464 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.210228 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:04.210239 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:04.212002 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:04.214091 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:04.214136 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:04.214139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:04.214475 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:04.215571 (+ 1096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:04.215573 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:04.215573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:04.215747 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:04.216077 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:04.216084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:04.216270 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 05:53:04.216384 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 05:53:04.216573 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1252,"thread_start_us":101,"threads_started":1}
I20260926 05:53:04.217453 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.203428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:53227 (local address 127.24.255.62:41143)
0926 05:53:04.205306 (+ 1878us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:04.205308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:04.205315 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:53:04.205340 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:04.205341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:04.205354 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:04.205410 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:04.206280 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.207240 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.208860 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.209183 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.209240 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.212139 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:04.212142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:04.212142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:04.212151 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:04.213971 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:04.214575 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:04.214576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:04.214577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:04.214645 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:04.215830 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:04.215831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:04.215832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:04.215987 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:04.216050 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:04.217250 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0926 05:53:04.217351 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1853,"thread_start_us":80,"threads_started":1}
I20260926 05:53:04.218261 4565 heartbeater.cc:347] Connected to a master server at 127.24.255.62:41143
I20260926 05:53:04.218364 4565 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:04.218585 4565 heartbeater.cc:511] Master 127.24.255.62:41143 requested a full tablet report, sending...
I20260926 05:53:04.219215 4353 ts_manager.cc:194] Registered new tserver with Master: 836f18151e3448148ff9ba2d06f70ae5 (127.24.255.1:37607)
I20260926 05:53:04.220124 4353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:53227
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:53:04.246531 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.255.20:59827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:41143
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:53:04 dist-test-slave-7fr8 krb5kdc[3876](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790401984, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 05:53:04.348891 4573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:04.349263 4573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:04.349304 4573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:04.351156 4573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:04.351274 4573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:04.351341 4573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:53:04.353834 4573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.255.20:59827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:04.354313 4573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:04.354676 4573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:53:04.362393 4581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:04.362393 4578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:04.362709 4579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:04.363382 4573 server_base.cc:1034] running on GCE node
I20260926 05:53:04.363701 4573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:04.363974 4573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:04.365163 4573 hybrid_clock.cc:648] HybridClock initialized: now 1790401984365130 us; error 45 us; skew 500 ppm
Sep 26 05:53:04 dist-test-slave-7fr8 krb5kdc[3876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401984, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:04.367813 4573 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:53:04.368295 4573 webserver.cc:460] Webserver started at http://127.24.255.2:40857/ using document root <none> and password file <none>
I20260926 05:53:04.368698 4573 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:04.368813 4573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:53:04.368981 4573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:53:04 dist-test-slave-7fr8 krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401984, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:04.382870 4573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "95d9c532d8e842b0aed0cc55ce1a1c49"
format_stamp: "Formatted at 2026-09-26 05:53:04 on dist-test-slave-7fr8"
server_key: "ee050f1454fda83d37598bbd84a1d303"
server_key_iv: "9d1962c613d5f148bea18aeff81a07ff"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:04.383355 4573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "95d9c532d8e842b0aed0cc55ce1a1c49"
format_stamp: "Formatted at 2026-09-26 05:53:04 on dist-test-slave-7fr8"
server_key: "ee050f1454fda83d37598bbd84a1d303"
server_key_iv: "9d1962c613d5f148bea18aeff81a07ff"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:04.385232 4573 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 05:53:04.402328 4589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:04.403314 4573 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 05:53:04.403448 4573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "95d9c532d8e842b0aed0cc55ce1a1c49"
format_stamp: "Formatted at 2026-09-26 05:53:04 on dist-test-slave-7fr8"
server_key: "ee050f1454fda83d37598bbd84a1d303"
server_key_iv: "9d1962c613d5f148bea18aeff81a07ff"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:04.403571 4573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:04.441699 4573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:04.444146 4573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:04.444363 4573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:04.444682 4573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:53:04.445111 4573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:53:04.445200 4573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:04.445276 4573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:53:04.445330 4573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:04.453701 4573 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:33045
I20260926 05:53:04.454272 4573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:53:04.454370 4702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:33045 every 8 connection(s)
I20260926 05:53:04.456450 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 4573
I20260926 05:53:04.456548 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:53:04.456722 25596 external_mini_cluster.cc:1428] Setting key c42f253e7ed782171d73a197ae8bf929
Sep 26 05:53:04 dist-test-slave-7fr8 krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401984, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:53:04.472700 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.455305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:37789 (local address 127.24.255.62:41143)
0926 05:53:04.455357 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:04.455359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:04.455635 (+ 276us) server_negotiation.cc:407] Connection header received
0926 05:53:04.462614 (+ 6979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:04.462616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:04.462632 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:04.462662 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:04.463587 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.464365 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.466031 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.466390 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.466442 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.469215 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:04.469217 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:04.469218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:04.469227 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:04.470976 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:04.471525 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:04.471527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:04.471527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:04.471593 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:04.471890 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:04.471891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:04.471892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:04.472041 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:04.472085 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:04.472484 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 05:53:04.472599 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:53:04.473268 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.455314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:41143 (local address 127.24.255.2:37789)
0926 05:53:04.455489 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:04.455547 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:04.462460 (+ 6913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:04.462888 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:04.462893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:04.463168 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:04.463467 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.463475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.464490 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.464491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.465909 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.465914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.466555 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.466556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.466714 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.467326 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:04.467337 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:04.469094 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:04.471079 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:04.471086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:04.471089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:04.471408 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:04.471667 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:04.471667 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:04.471668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:04.471808 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:04.472134 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:04.472139 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:04.472317 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 05:53:04.472848 (+ 531us) client_negotiation.cc:240] Negotiation successful
0926 05:53:04.473026 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260926 05:53:04.474174 4703 heartbeater.cc:347] Connected to a master server at 127.24.255.62:41143
I20260926 05:53:04.474273 4703 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:04.474462 4703 heartbeater.cc:511] Master 127.24.255.62:41143 requested a full tablet report, sending...
I20260926 05:53:04.474911 4353 ts_manager.cc:194] Registered new tserver with Master: 95d9c532d8e842b0aed0cc55ce1a1c49 (127.24.255.2:33045)
I20260926 05:53:04.475384 4353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:37789
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:53:04.503079 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.255.20:59827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:41143
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:53:04.635121 4708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:04.635357 4708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:04.635391 4708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:04.636832 4708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:04.636890 4708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:04.636930 4708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:53:04.639230 4708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.255.20:59827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:41143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:04.639532 4708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:04.639806 4708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:53:04.645642 4714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:04.645839 4716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:04.645642 4713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:04.646232 4708 server_base.cc:1034] running on GCE node
I20260926 05:53:04.646443 4708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:04.646721 4708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:04.647908 4708 hybrid_clock.cc:648] HybridClock initialized: now 1790401984647867 us; error 49 us; skew 500 ppm
Sep 26 05:53:04 dist-test-slave-7fr8 krb5kdc[3876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401984, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:04.651508 4708 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:53:04.652019 4708 webserver.cc:460] Webserver started at http://127.24.255.3:45225/ using document root <none> and password file <none>
I20260926 05:53:04.652251 4708 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:04.652308 4708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:53:04.652447 4708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:53:04 dist-test-slave-7fr8 krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401984, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:04.680580 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "51a23f06592742198e474e030ecc8d21"
format_stamp: "Formatted at 2026-09-26 05:53:04 on dist-test-slave-7fr8"
server_key: "39c280ad5eee4c8c49cfa94aa5f38ba6"
server_key_iv: "0bdc6c98861998de9c07be8abbf72e8c"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:04.681519 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "51a23f06592742198e474e030ecc8d21"
format_stamp: "Formatted at 2026-09-26 05:53:04 on dist-test-slave-7fr8"
server_key: "39c280ad5eee4c8c49cfa94aa5f38ba6"
server_key_iv: "0bdc6c98861998de9c07be8abbf72e8c"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:04.683570 4708 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 05:53:04.702272 4723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:04.702564 4708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 05:53:04.702662 4708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "51a23f06592742198e474e030ecc8d21"
format_stamp: "Formatted at 2026-09-26 05:53:04 on dist-test-slave-7fr8"
server_key: "39c280ad5eee4c8c49cfa94aa5f38ba6"
server_key_iv: "0bdc6c98861998de9c07be8abbf72e8c"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:04.702769 4708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:04.715924 4708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:04.718490 4708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:04.718714 4708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:04.718979 4708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:53:04.719470 4708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:53:04.719573 4708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:04.719674 4708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:53:04.719759 4708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:04.728868 4708 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:33421
I20260926 05:53:04.729430 4708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:53:04.731796 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 4708
I20260926 05:53:04.731894 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:53:04.732084 25596 external_mini_cluster.cc:1428] Setting key 13e8aa8774c466a663e583608fd9a18c
I20260926 05:53:04.733265 4836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:33421 every 8 connection(s)
Sep 26 05:53:04 dist-test-slave-7fr8 krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401984, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:53:04.745538 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.732440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:40485 (local address 127.24.255.62:41143)
0926 05:53:04.732587 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:04.732589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:04.734121 (+ 1532us) server_negotiation.cc:407] Connection header received
0926 05:53:04.734826 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:04.734827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:04.734839 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:04.734869 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:04.735745 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.736494 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.737991 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.738327 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.738376 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.741114 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:04.741116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:04.741117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:04.741125 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:04.743083 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:04.744354 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:04.744356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:04.744356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:04.744421 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:04.744713 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:04.744714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:04.744714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:04.744872 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:04.744919 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:04.745337 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 05:53:04.745433 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260926 05:53:04.746424 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.731374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:41143 (local address 127.24.255.3:40485)
0926 05:53:04.733951 (+ 2577us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:04.733998 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:04.734701 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:04.735073 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:04.735077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:04.735341 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:04.735635 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.735642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.736609 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.736610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.737883 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.737887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.738479 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.738480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.738642 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.739215 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:04.739226 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:04.740968 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:04.743856 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:04.743869 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:04.743872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:04.744257 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:04.744491 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:04.744491 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:04.744492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:04.744635 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:04.744968 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:04.744975 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:04.745178 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 05:53:04.745926 (+ 748us) client_negotiation.cc:240] Negotiation successful
0926 05:53:04.746164 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2530,"thread_start_us":98,"threads_started":1}
I20260926 05:53:04.747215 4837 heartbeater.cc:347] Connected to a master server at 127.24.255.62:41143
I20260926 05:53:04.747321 4837 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:04.747529 4837 heartbeater.cc:511] Master 127.24.255.62:41143 requested a full tablet report, sending...
I20260926 05:53:04.748035 4353 ts_manager.cc:194] Registered new tserver with Master: 51a23f06592742198e474e030ecc8d21 (127.24.255.3:33421)
I20260926 05:53:04.748553 4353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:40485
I20260926 05:53:04.755367 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:53:04.773810 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.762504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.24.255.62:41143)
0926 05:53:04.762613 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:04.762615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:04.765573 (+ 2958us) server_negotiation.cc:407] Connection header received
0926 05:53:04.765688 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:04.765690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:04.765702 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:04.765732 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:04.766831 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.767638 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.768798 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.769089 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.769138 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.770837 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:04.770839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:04.770840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:04.770849 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:04.772479 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:04.772905 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:04.772907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:04.772908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:04.772967 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:04.773190 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:04.773191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:04.773192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:04.773337 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:04.773410 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:04.773625 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 05:53:04.773710 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 05:53:04.776320 4353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:53:04.778839 4353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:53:04.803882 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.797139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:33045 (local address 127.0.0.1:34164)
0926 05:53:04.798170 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:04.798215 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:04.798283 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:04.798881 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:04.798884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:04.798887 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:53:04.798967 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.798969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.800616 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.800617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.802405 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.802409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.803545 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.803546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.803639 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.803653 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 05:53:04.803724 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 05:53:04.803788 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"thread_start_us":102,"threads_started":1}
I20260926 05:53:04.807832 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.797331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.24.255.2:33045)
0926 05:53:04.798648 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:04.798654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:04.798664 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:53:04.798703 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:04.798705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:04.798720 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:04.798790 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:53:04.799067 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.800516 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.802512 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.803337 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.803413 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.803504 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:04.807631 (+ 4127us) server_negotiation.cc:299] Negotiation successful
0926 05:53:04.807691 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1277,"thread_start_us":76,"threads_started":1}
I20260926 05:53:04.808981 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 4251bd37a67d4635a7a468b04d68d9f4 (DEFAULT_TABLE table=test-table [id=0b60b9cc29ab452ba0550ebe8c71d382]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:53:04.809389 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4251bd37a67d4635a7a468b04d68d9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:53:04.810273 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.797141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:37607 (local address 127.0.0.1:40088)
0926 05:53:04.798391 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:04.798400 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:04.798435 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:04.804652 (+ 6217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:04.804654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:04.804656 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:53:04.804702 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.804704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.806312 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.806314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.807507 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.807511 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.809971 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.809972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.810052 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.810058 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:53:04.810138 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 05:53:04.810189 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1233,"thread_start_us":74,"threads_started":1}
I20260926 05:53:04.810784 4854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.797497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.24.255.1:37607)
0926 05:53:04.804428 (+ 6931us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:04.804433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:04.804444 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:53:04.804480 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:04.804483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:04.804502 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:04.804578 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:53:04.804774 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.806211 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.808874 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.809806 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.809878 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.809933 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:04.810662 (+ 729us) server_negotiation.cc:299] Negotiation successful
0926 05:53:04.810716 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6889,"thread_start_us":50,"threads_started":1}
I20260926 05:53:04.811539 4499 tablet_service.cc:1467] Processing CreateTablet for tablet 4251bd37a67d4635a7a468b04d68d9f4 (DEFAULT_TABLE table=test-table [id=0b60b9cc29ab452ba0550ebe8c71d382]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:53:04.811860 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4251bd37a67d4635a7a468b04d68d9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:53:04.813450 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.797370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:33421 (local address 127.0.0.1:52280)
0926 05:53:04.798499 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:04.798506 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:04.798538 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:04.809745 (+ 11207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:04.809746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:04.809749 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:53:04.809801 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.809803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.811172 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.811173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.812264 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.812267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.813226 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.813227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.813290 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.813296 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:53:04.813323 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 05:53:04.813365 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"thread_start_us":65,"threads_started":1}
I20260926 05:53:04.813843 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.797562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.24.255.3:33421)
0926 05:53:04.809558 (+ 11996us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:04.809563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:04.809576 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:53:04.809615 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:04.809618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:04.809634 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:04.809863 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:53:04.809932 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.811086 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.812386 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.813128 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.813614 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.813703 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:04.813728 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 05:53:04.813770 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11956,"thread_start_us":87,"threads_started":1}
I20260926 05:53:04.814440 4771 tablet_service.cc:1467] Processing CreateTablet for tablet 4251bd37a67d4635a7a468b04d68d9f4 (DEFAULT_TABLE table=test-table [id=0b60b9cc29ab452ba0550ebe8c71d382]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:53:04.814747 4771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4251bd37a67d4635a7a468b04d68d9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:53:04.816246 4857 tablet_bootstrap.cc:492] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5: Bootstrap starting.
I20260926 05:53:04.816982 4857 tablet_bootstrap.cc:654] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5: Neither blocks nor log segments found. Creating new log.
I20260926 05:53:04.817350 4857 log.cc:824] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5: Log is configured to *not* fsync() on all Append() calls
I20260926 05:53:04.818563 4857 tablet_bootstrap.cc:492] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5: No bootstrap required, opened a new log
I20260926 05:53:04.818715 4857 ts_tablet_manager.cc:1397] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:53:04.821276 4856 tablet_bootstrap.cc:492] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49: Bootstrap starting.
I20260926 05:53:04.821592 4859 tablet_bootstrap.cc:492] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21: Bootstrap starting.
I20260926 05:53:04.822476 4856 tablet_bootstrap.cc:654] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49: Neither blocks nor log segments found. Creating new log.
I20260926 05:53:04.822800 4856 log.cc:824] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49: Log is configured to *not* fsync() on all Append() calls
I20260926 05:53:04.823604 4859 tablet_bootstrap.cc:654] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21: Neither blocks nor log segments found. Creating new log.
I20260926 05:53:04.822758 4857 raft_consensus.cc:348] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 33045 } } peers { permanent_uuid: "51a23f06592742198e474e030ecc8d21" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33421 } } peers { permanent_uuid: "836f18151e3448148ff9ba2d06f70ae5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 37607 } }
I20260926 05:53:04.823874 4857 raft_consensus.cc:374] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:53:04.823958 4857 raft_consensus.cc:729] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836f18151e3448148ff9ba2d06f70ae5, State: Initialized, Role: FOLLOWER
I20260926 05:53:04.824174 4857 consensus_queue.cc:260] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 33045 } } peers { permanent_uuid: "51a23f06592742198e474e030ecc8d21" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33421 } } peers { permanent_uuid: "836f18151e3448148ff9ba2d06f70ae5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 37607 } }
I20260926 05:53:04.825071 4857 ts_tablet_manager.cc:1428] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260926 05:53:04.825448 4859 log.cc:824] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21: Log is configured to *not* fsync() on all Append() calls
I20260926 05:53:04.825846 4565 heartbeater.cc:503] Master 127.24.255.62:41143 was elected leader, sending a full tablet report...
I20260926 05:53:04.827014 4859 tablet_bootstrap.cc:492] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21: No bootstrap required, opened a new log
I20260926 05:53:04.827124 4859 ts_tablet_manager.cc:1397] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260926 05:53:04.829499 4859 raft_consensus.cc:348] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 33045 } } peers { permanent_uuid: "51a23f06592742198e474e030ecc8d21" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33421 } } peers { permanent_uuid: "836f18151e3448148ff9ba2d06f70ae5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 37607 } }
I20260926 05:53:04.829756 4859 raft_consensus.cc:374] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:53:04.829826 4859 raft_consensus.cc:729] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a23f06592742198e474e030ecc8d21, State: Initialized, Role: FOLLOWER
I20260926 05:53:04.830022 4859 consensus_queue.cc:260] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 33045 } } peers { permanent_uuid: "51a23f06592742198e474e030ecc8d21" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33421 } } peers { permanent_uuid: "836f18151e3448148ff9ba2d06f70ae5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 37607 } }
I20260926 05:53:04.830392 4859 ts_tablet_manager.cc:1428] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:53:04.830555 4837 heartbeater.cc:503] Master 127.24.255.62:41143 was elected leader, sending a full tablet report...
I20260926 05:53:04.832420 4856 tablet_bootstrap.cc:492] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49: No bootstrap required, opened a new log
I20260926 05:53:04.832500 4856 ts_tablet_manager.cc:1397] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260926 05:53:04.835155 4856 raft_consensus.cc:348] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 33045 } } peers { permanent_uuid: "51a23f06592742198e474e030ecc8d21" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33421 } } peers { permanent_uuid: "836f18151e3448148ff9ba2d06f70ae5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 37607 } }
I20260926 05:53:04.835891 4856 raft_consensus.cc:374] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:53:04.835978 4856 raft_consensus.cc:729] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d9c532d8e842b0aed0cc55ce1a1c49, State: Initialized, Role: FOLLOWER
I20260926 05:53:04.836180 4856 consensus_queue.cc:260] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 33045 } } peers { permanent_uuid: "51a23f06592742198e474e030ecc8d21" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33421 } } peers { permanent_uuid: "836f18151e3448148ff9ba2d06f70ae5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 37607 } }
I20260926 05:53:04.836532 4856 ts_tablet_manager.cc:1428] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 05:53:04.836685 4703 heartbeater.cc:503] Master 127.24.255.62:41143 was elected leader, sending a full tablet report...
Sep 26 05:53:04 dist-test-slave-7fr8 krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401984, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:04.884459 4864 raft_consensus.cc:482] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:53:04.884634 4864 raft_consensus.cc:504] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 33045 } } peers { permanent_uuid: "51a23f06592742198e474e030ecc8d21" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33421 } } peers { permanent_uuid: "836f18151e3448148ff9ba2d06f70ae5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 37607 } }
I20260926 05:53:04.885012 4864 leader_election.cc:288] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51a23f06592742198e474e030ecc8d21 (127.24.255.3:33421), 836f18151e3448148ff9ba2d06f70ae5 (127.24.255.1:37607)
I20260926 05:53:04.889281 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.885272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:37607 (local address 127.24.255.2:55435)
0926 05:53:04.885395 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:04.885410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:04.885456 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:04.885720 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:04.885722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:04.885724 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:53:04.885791 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.885793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.887054 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.887055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.888218 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.888222 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.889019 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.889020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.889099 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.889103 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:53:04.889145 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 05:53:04.889193 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 05:53:04.890045 4854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.885526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:55435 (local address 127.24.255.1:37607)
0926 05:53:04.885571 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:04.885573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:04.885582 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:53:04.885609 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:04.885610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:04.885618 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:04.885670 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:53:04.886223 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.886946 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.888331 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.888903 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.888973 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.888983 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:04.889923 (+ 940us) server_negotiation.cc:299] Negotiation successful
0926 05:53:04.889975 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:53:04.890460 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4251bd37a67d4635a7a468b04d68d9f4" candidate_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836f18151e3448148ff9ba2d06f70ae5" is_pre_election: true
I20260926 05:53:04.890626 4519 raft_consensus.cc:2454] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d9c532d8e842b0aed0cc55ce1a1c49 in term 0.
I20260926 05:53:04.891085 4590 leader_election.cc:302] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 836f18151e3448148ff9ba2d06f70ae5, 95d9c532d8e842b0aed0cc55ce1a1c49; no voters:
I20260926 05:53:04.891309 4864 raft_consensus.cc:2781] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:53:04.891359 4864 raft_consensus.cc:482] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:53:04.891379 4864 raft_consensus.cc:3037] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:53:04.892289 4864 raft_consensus.cc:504] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 33045 } } peers { permanent_uuid: "51a23f06592742198e474e030ecc8d21" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33421 } } peers { permanent_uuid: "836f18151e3448148ff9ba2d06f70ae5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 37607 } }
I20260926 05:53:04.892444 4864 leader_election.cc:288] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [CANDIDATE]: Term 1 election: Requested vote from peers 51a23f06592742198e474e030ecc8d21 (127.24.255.3:33421), 836f18151e3448148ff9ba2d06f70ae5 (127.24.255.1:37607)
I20260926 05:53:04.892792 4519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4251bd37a67d4635a7a468b04d68d9f4" candidate_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836f18151e3448148ff9ba2d06f70ae5"
I20260926 05:53:04.892875 4519 raft_consensus.cc:3037] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:53:04.893764 4519 raft_consensus.cc:2454] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d9c532d8e842b0aed0cc55ce1a1c49 in term 1.
I20260926 05:53:04.894021 4590 leader_election.cc:302] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 836f18151e3448148ff9ba2d06f70ae5, 95d9c532d8e842b0aed0cc55ce1a1c49; no voters:
I20260926 05:53:04.894176 4864 raft_consensus.cc:2781] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:53:04.894333 4864 raft_consensus.cc:686] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [term 1 LEADER]: Becoming Leader. State: Replica: 95d9c532d8e842b0aed0cc55ce1a1c49, State: Running, Role: LEADER
I20260926 05:53:04.894434 4864 consensus_queue.cc:237] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 33045 } } peers { permanent_uuid: "51a23f06592742198e474e030ecc8d21" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33421 } } peers { permanent_uuid: "836f18151e3448148ff9ba2d06f70ae5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 37607 } }
I20260926 05:53:04.895288 4353 catalog_manager.cc:5495] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95d9c532d8e842b0aed0cc55ce1a1c49 (127.24.255.2). New cstate: current_term: 1 leader_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 33045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51a23f06592742198e474e030ecc8d21" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836f18151e3448148ff9ba2d06f70ae5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 37607 } health_report { overall_health: UNKNOWN } } }
I20260926 05:53:04.899679 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.885930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:33421 (local address 127.24.255.2:33921)
0926 05:53:04.889371 (+ 3441us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:04.889382 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:04.889419 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:04.896388 (+ 6969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:04.896390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:04.896392 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:53:04.896459 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.896461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.897371 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.897373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.898648 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:04.898652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.899448 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.899449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:04.899513 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.899517 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:53:04.899552 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 05:53:04.899595 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3429,"thread_start_us":97,"threads_started":1}
I20260926 05:53:04.900069 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.886145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:33921 (local address 127.24.255.3:33421)
0926 05:53:04.886189 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:04.886191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:04.889470 (+ 3279us) server_negotiation.cc:407] Connection header received
0926 05:53:04.889511 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:04.889512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:04.889522 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:04.889561 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:53:04.896552 (+ 6991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.897255 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.898747 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.899302 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.899366 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.899377 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:04.899937 (+ 560us) server_negotiation.cc:299] Negotiation successful
0926 05:53:04.899986 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:53:04.900719 4789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4251bd37a67d4635a7a468b04d68d9f4" candidate_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a23f06592742198e474e030ecc8d21" is_pre_election: true
I20260926 05:53:04.900876 4789 raft_consensus.cc:2454] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d9c532d8e842b0aed0cc55ce1a1c49 in term 0.
I20260926 05:53:04.900978 4791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4251bd37a67d4635a7a468b04d68d9f4" candidate_uuid: "95d9c532d8e842b0aed0cc55ce1a1c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a23f06592742198e474e030ecc8d21"
I20260926 05:53:04.901026 4791 raft_consensus.cc:3037] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:53:04.901909 4791 raft_consensus.cc:2454] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d9c532d8e842b0aed0cc55ce1a1c49 in term 1.
I20260926 05:53:04.913790 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.910593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.24.255.2:33045)
0926 05:53:04.910640 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:04.910642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:04.910648 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:53:04.910688 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:04.910689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:04.910709 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:04.910773 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:53:04.910983 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.911665 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.912699 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:04.913003 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:04.913059 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:04.913238 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:53:04.913375 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:53:04.913441 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:04.913670 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 05:53:04.913691 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:53:04.916354 4519 raft_consensus.cc:1261] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5 [term 1 FOLLOWER]: Refusing update from remote peer 95d9c532d8e842b0aed0cc55ce1a1c49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:53:04.916877 4866 consensus_queue.cc:1035] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836f18151e3448148ff9ba2d06f70ae5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 37607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:53:04.920181 4791 raft_consensus.cc:1261] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21 [term 1 FOLLOWER]: Refusing update from remote peer 95d9c532d8e842b0aed0cc55ce1a1c49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:53:04.920589 4866 consensus_queue.cc:1035] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a23f06592742198e474e030ecc8d21" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 33421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:53:04.924675 4869 mvcc.cc:205] Tried to move back new op lower bound from 7333486530212454400 to 7333486530128330752. Current Snapshot: MvccSnapshot[applied={T|T < 7333486530212454400}]
I20260926 05:53:04.925340 4872 mvcc.cc:205] Tried to move back new op lower bound from 7333486530212454400 to 7333486530128330752. Current Snapshot: MvccSnapshot[applied={T|T < 7333486530212454400}]
I20260926 05:53:04.937844 4353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:53:04.937989 4353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:53:04.938982 4353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 757168f6ab564365b3ed9d12fa651fdc: Sending DeleteTablet for 3 replicas of tablet 4251bd37a67d4635a7a468b04d68d9f4
I20260926 05:53:04.939477 4499 tablet_service.cc:1514] Processing DeleteTablet for tablet 4251bd37a67d4635a7a468b04d68d9f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:53:04 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:40088
I20260926 05:53:04.939811 4879 tablet_replica.cc:330] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5: stopping tablet replica
I20260926 05:53:04.939883 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 4251bd37a67d4635a7a468b04d68d9f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:53:04 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:34164
I20260926 05:53:04.939991 4879 raft_consensus.cc:2229] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:53:04.940115 4879 raft_consensus.cc:2258] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:53:04.940111 4771 tablet_service.cc:1514] Processing DeleteTablet for tablet 4251bd37a67d4635a7a468b04d68d9f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:53:04 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:52280
I20260926 05:53:04.940578 4879 ts_tablet_manager.cc:1905] T 4251bd37a67d4635a7a468b04d68d9f4 P 836f18151e3448148ff9ba2d06f70ae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:53:04.940764 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 4434
I20260926 05:53:04.944610 4881 tablet_replica.cc:330] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21: stopping tablet replica
I20260926 05:53:04.944836 4880 tablet_replica.cc:330] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49: stopping tablet replica
I20260926 05:53:04.945039 4880 raft_consensus.cc:2229] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:53:04.945194 4881 raft_consensus.cc:2229] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:53:04.945401 4880 raft_consensus.cc:2258] T 4251bd37a67d4635a7a468b04d68d9f4 P 95d9c532d8e842b0aed0cc55ce1a1c49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:53:04.945554 4881 raft_consensus.cc:2258] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 05:53:04.945927 4337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:37607 (error 108)
I20260926 05:53:04.946533 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 4573
I20260926 05:53:04.946561 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.946441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:37607 (local address 127.0.0.1:40104)
0926 05:53:04.946492 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:04.946524 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:37607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 05:53:04.946657 4337 catalog_manager.cc:4584] TS 836f18151e3448148ff9ba2d06f70ae5 (127.24.255.1:37607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4251bd37a67d4635a7a468b04d68d9f4: Network error: Client connection negotiation failed: client connection to 127.24.255.1:37607: connect: Connection refused (error 111)
I20260926 05:53:04.950294 4881 ts_tablet_manager.cc:1905] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:53:04.956619 4881 ts_tablet_manager.cc:1918] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:53:04.956816 4881 log.cc:1196] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/4251bd37a67d4635a7a468b04d68d9f4
I20260926 05:53:04.957460 4881 ts_tablet_manager.cc:1939] T 4251bd37a67d4635a7a468b04d68d9f4 P 51a23f06592742198e474e030ecc8d21: Deleting consensus metadata
I20260926 05:53:04.958150 4340 catalog_manager.cc:4841] TS 51a23f06592742198e474e030ecc8d21 (127.24.255.3:33421): tablet 4251bd37a67d4635a7a468b04d68d9f4 (table test-table [id=0b60b9cc29ab452ba0550ebe8c71d382]) successfully deleted
I20260926 05:53:04.958969 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:04.958845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:33045 (local address 127.0.0.1:34182)
0926 05:53:04.958902 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:04.958932 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.2:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 05:53:04.959182 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 4708
I20260926 05:53:04.970212 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 4321
I20260926 05:53:04.977187 25596 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 05:53:05.322899 25596 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 05:53:05.323020 25596 mini_ranger.cc:66] Stopping Ranger...
I20260926 05:53:05.658809 25596 mini_ranger.cc:68] Stopped Ranger
2026-09-26 05:53:05.659 UTC [3906] LOG: received smart shutdown request
2026-09-26 05:53:05.663 UTC [3906] LOG: background worker "logical replication launcher" (PID 3915) exited with exit code 1
2026-09-26 05:53:05.663 UTC [3910] LOG: shutting down
2026-09-26 05:53:05.892 UTC [3906] LOG: database system is shut down
2026-09-26T05:53:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49218 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:53:06 dist-test-slave-7fr8 krb5kdc[4891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:53:06 dist-test-slave-7fr8 krb5kdc[4891](info): set up 2 sockets
Sep 26 05:53:06 dist-test-slave-7fr8 krb5kdc[4891](info): commencing operation
krb5kdc: starting...
W20260926 05:53:06.974433 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:53:07 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:53:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:53:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.255.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.255.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/rangeradmin_127.24.255.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/rangeradmin_127.24.255.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.255.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.255.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/rangerlookup_127.24.255.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/rangerlookup_127.24.255.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.255.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.255.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/HTTP_127.24.255.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/HTTP_127.24.255.20@KRBTEST.COM.keytab.
I20260926 05:53:07.149734 25596 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGUnhcE/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/postgres -l logfile start
2026-09-26 05:53:10.016 UTC [4921] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 05:53:10.016 UTC [4921] LOG: listening on IPv4 address "127.24.255.20", port 35365
2026-09-26 05:53:10.021 UTC [4921] LOG: listening on Unix socket "/tmp/.s.PGSQL.35365"
2026-09-26 05:53:10.053 UTC [4924] LOG: database system was shut down at 2026-09-26 05:53:08 UTC
2026-09-26 05:53:10.058 UTC [4921] LOG: database system is ready to accept connections
I20260926 05:53:10.926117 25596 mini_postgres.cc:96] Postgres bound to 35365
2026-09-26 05:53:10.931 UTC [4932] FATAL: database "slave" does not exist
127.24.255.20:35365 - accepting connections
I20260926 05:53:10.932662 25596 mini_ranger.cc:161] Starting Ranger...
I20260926 05:53:10.948312 25596 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 05:53:11.512946 25596 mini_ranger.cc:87] Created ranger Postgres database
I20260926 05:53:11.513037 25596 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-admin
2026-09-26 05:53:11,797 [I] DB FLAVOR :POSTGRES
2026-09-26 05:53:11,798 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 05:53:11,798 [I] Checking connection..
2026-09-26 05:53:11,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 05:53:12,073 [I] Checking connection passed.
2026-09-26 05:53:12,073 [I] --------- Verifying version history table ---------
2026-09-26 05:53:12,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:53:12,348 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 05:53:12,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:53:12,597 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 05:53:12,598 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 05:53:12,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 05:53:12.855 UTC [4996] WARNING: there is no transaction in progress
2026-09-26 05:53:12,861 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 05:53:12,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:53:13,131 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 05:53:13,131 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 05:53:13,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 05:53:13,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 05:53:13,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7fr8.c.gcp-upstream.internal','N') ;"
2026-09-26 05:53:13,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 05:53:14,152 [I] Table x_portal_user does not exist in database ranger
2026-09-26 05:53:14,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 05:53:14,421 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 05:53:14,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 05:53:14,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 05:53:14,949 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 05:53:14,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 05:53:16.002 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.020 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.039 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.058 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.078 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.098 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.160 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.168 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.185 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.197 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.209 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.225 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.237 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.244 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.252 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:16.261 UTC [5104] WARNING: there is no transaction in progress
2026-09-26 05:53:17,019 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 05:53:17,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 05:53:17,259 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 05:53:17,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 05:53:17,521 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 05:53:17,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 05:53:17,774 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 05:53:17,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 05:53:18,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 05:53:18,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7fr8.c.gcp-upstream.internal';"
2026-09-26 05:53:18,538 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 05:53:18,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 05:53:18,823 [I] DB_PATCHES have already been applied
I20260926 05:53:18.831175 25596 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGUnhcE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 05:53:18.831269 25596 mini_ranger.cc:193] Using host: 127.24.255.20
I20260926 05:53:18.834887 25596 mini_ranger.cc:227] Ranger admin URL: http://127.24.255.20:56317
Sep 26, 2026 5:53:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 5:53:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 5:53:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 5:53:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 5:53:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 5:53:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 5:53:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 05:53:19.850992 25596 mini_ranger.cc:160] Time spent starting Ranger: real 8.918s user 0.002s sys 0.008s
Sep 26, 2026 5:53:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 5:53:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 5:53:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 5:53:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 5:53:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 5:53:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 5:53:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 5:53:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 5:53:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 5:53:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 5:53:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 5:53:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 5:53:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 5:53:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 5:53:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 5:53:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 5:53:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 5:53:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 05:53:43.935514 25596 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.255.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.255.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/rangerkms_127.24.255.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.255.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/rangerkms_127.24.255.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.255.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.255.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.255.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/HTTP_127.24.255.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.255.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/HTTP_127.24.255.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 05:53:44 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402024, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 05:53:44.037025 25596 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 05:53:44.048732 25596 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 05:53:45.035557 25596 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 05:53:45.035631 25596 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-kms
I20260926 05:53:45.035643 25596 mini_ranger_kms.cc:227] Using postgres at 127.24.255.20:35365
2026-09-26 05:53:45,191 [I] DB FLAVOR :POSTGRES
2026-09-26 05:53:45,192 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 05:53:45,192 [I] Checking connection
2026-09-26 05:53:45,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 05:53:45,448 [I] connection success
2026-09-26 05:53:45,448 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 05:53:45,448 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 05:53:45,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 05:53:45,738 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 05:53:45,738 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 05:53:45,739 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 05:53:45,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.255.20:35365/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 05:53:46,053 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 05:53:46.524237 25596 mini_ranger_kms.cc:310] Created kms service
I20260926 05:53:46.702785 25596 mini_ranger_kms.cc:326] Created kudu user
I20260926 05:53:46.745934 25596 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 05:53:46 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402024, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:47.159798 25596 mini_ranger_kms.cc:384] Added ranger policy
I20260926 05:53:47.159937 25596 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGUnhcE/build/release/bin/postgresql.jar:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGUnhcE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 05:53:47.159956 25596 mini_ranger_kms.cc:240] Using host: 127.24.255.20
I20260926 05:53:47.163245 25596 mini_ranger_kms.cc:276] Ranger KMS PID: 5285
I20260926 05:53:47.163328 25596 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.255.20:40209
Sep 26, 2026 5:53:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 5:53:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 5:53:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 5:53:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 5:53:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 5:53:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 05:53:49.266358 25596 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.229s user 0.010s sys 0.128s
I20260926 05:53:49.266471 25596 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 05:53:49.266518 25596 mini_ranger_kms.cc:401] 127.24.255.20:40209/kms/v1/keys
Sep 26 05:53:49 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790402029, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.255.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 05:53:49 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402029, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.255.20@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
Sep 26 05:53:50 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402029, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.255.20@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
Sep 26 05:53:52 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:53:52.762661 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:34717
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:34717
--ranger_config_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.255.20:40209/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:53:52.849076 5335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:52.849304 5335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:52.849326 5335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:52.850621 5335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:53:52.850670 5335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:52.850683 5335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:52.850694 5335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:53:52.850705 5335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:53:52.853252 5335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.255.20:40209/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:34717
--ranger_config_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:34717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:52.853555 5335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:52.853808 5335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:53:52.859483 5341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:52.859483 5340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:52.859549 5343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:52.859871 5335 server_base.cc:1034] running on GCE node
I20260926 05:53:52.860139 5335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:52.860435 5335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:52.861613 5335 hybrid_clock.cc:648] HybridClock initialized: now 1790402032861592 us; error 33 us; skew 500 ppm
Sep 26 05:53:52 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402032, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:52.864568 5335 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:53:52.865060 5335 webserver.cc:460] Webserver started at http://127.24.255.62:43171/ using document root <none> and password file <none>
I20260926 05:53:52.865268 5335 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:52.865309 5335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:53:52.865437 5335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:53:52 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402032, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:52.914809 5335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "a1fe3693d11d4625a95fd2269aa4f45d"
format_stamp: "Formatted at 2026-09-26 05:53:52 on dist-test-slave-7fr8"
server_key: "77b412d331e0ac4b55eb37fdb2a4f086"
server_key_iv: "fc7a4c6c5e8abd674e8c695d85e2d966"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:52.915288 5335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "a1fe3693d11d4625a95fd2269aa4f45d"
format_stamp: "Formatted at 2026-09-26 05:53:52 on dist-test-slave-7fr8"
server_key: "77b412d331e0ac4b55eb37fdb2a4f086"
server_key_iv: "fc7a4c6c5e8abd674e8c695d85e2d966"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:52.916664 5335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:53:52.929082 5350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:52.929288 5335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:53:52.929388 5335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "a1fe3693d11d4625a95fd2269aa4f45d"
format_stamp: "Formatted at 2026-09-26 05:53:52 on dist-test-slave-7fr8"
server_key: "77b412d331e0ac4b55eb37fdb2a4f086"
server_key_iv: "fc7a4c6c5e8abd674e8c695d85e2d966"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:52.929489 5335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:52.950573 5335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:52.952459 5335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:52.952625 5335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:52.956846 5335 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:34717
I20260926 05:53:52.956884 5402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:34717 every 8 connection(s)
I20260926 05:53:52.957264 5335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:53:52.957935 5403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:53:52.959182 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 5335
I20260926 05:53:52.959275 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:53:52.959470 25596 external_mini_cluster.cc:1428] Setting key 5d9e38f91bca86617fc11dd7988edaac
I20260926 05:53:52.961300 5403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d: Bootstrap starting.
I20260926 05:53:52.962056 5403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d: Neither blocks nor log segments found. Creating new log.
I20260926 05:53:52.962432 5403 log.cc:824] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:53:52.963351 5403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d: No bootstrap required, opened a new log
I20260926 05:53:52.965934 5403 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34717 } }
I20260926 05:53:52.966513 5403 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:53:52.966567 5403 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fe3693d11d4625a95fd2269aa4f45d, State: Initialized, Role: FOLLOWER
I20260926 05:53:52.966717 5403 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34717 } }
I20260926 05:53:52.966815 5403 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:53:52.966845 5403 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:53:52.966887 5403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:53:52.967603 5403 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34717 } }
I20260926 05:53:52.967715 5403 leader_election.cc:302] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1fe3693d11d4625a95fd2269aa4f45d; no voters:
I20260926 05:53:52.967914 5403 leader_election.cc:288] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:53:52.967950 5408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:53:52.968147 5408 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 1 LEADER]: Becoming Leader. State: Replica: a1fe3693d11d4625a95fd2269aa4f45d, State: Running, Role: LEADER
I20260926 05:53:52.968223 5403 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:53:52.968245 5408 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34717 } }
I20260926 05:53:52.968650 5409 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34717 } } }
I20260926 05:53:52.968741 5409 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [sys.catalog]: This master's current role is: LEADER
I20260926 05:53:52.968902 5410 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1fe3693d11d4625a95fd2269aa4f45d. Latest consensus state: current_term: 1 leader_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34717 } } }
I20260926 05:53:52.968964 5410 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [sys.catalog]: This master's current role is: LEADER
I20260926 05:53:52.969084 5416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:53:52.969617 5416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:53:52.969925 5403 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 26 05:53:52 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:53:52.975293 5416 catalog_manager.cc:1347] Generated new cluster ID: c49b4d6d372047aeb2d87a14ff84619f
I20260926 05:53:52.975443 5416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:53:52.978365 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:52.960252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49510 (local address 127.24.255.62:34717)
0926 05:53:52.962735 (+ 2483us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:52.962745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:52.962780 (+ 35us) server_negotiation.cc:407] Connection header received
0926 05:53:52.963387 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:52.963397 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:52.963463 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:52.963680 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:52.964486 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:52.965692 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:52.967126 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:52.967606 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:52.967704 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:52.972128 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:52.972154 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:52.972155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:52.972270 (+ 115us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:52.975960 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:52.976449 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:52.976453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:52.976453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:52.976569 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:52.976779 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:52.976780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:52.976780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:52.977109 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:52.977329 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:52.977899 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 05:53:52.978078 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2434,"thread_start_us":72,"threads_started":1}
I20260926 05:53:52.996609 5416 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:53:52.997288 5416 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:53:53.004691 5416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d: Generated new TSK 0
I20260926 05:53:53.004899 5416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:53:53.710737 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.255.20:40209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34717
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:53:53.830878 5448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:53.831219 5448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:53.831266 5448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:53.833384 5448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:53.833482 5448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:53.833528 5448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:53:53.836246 5448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.255.20:40209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:53.836599 5448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:53.836926 5448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:53:53.843665 5454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:53.843665 5456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:53.844614 5453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:53.844666 5448 server_base.cc:1034] running on GCE node
I20260926 05:53:53.844916 5448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:53.845186 5448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:53.846375 5448 hybrid_clock.cc:648] HybridClock initialized: now 1790402033846357 us; error 33 us; skew 500 ppm
Sep 26 05:53:53 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402033, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:53.849236 5448 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:53:53.849769 5448 webserver.cc:460] Webserver started at http://127.24.255.1:40565/ using document root <none> and password file <none>
I20260926 05:53:53.849998 5448 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:53.850047 5448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:53:53.850222 5448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:53:53 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402033, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:53.872100 5448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "fe8ceab0a7e0482d9544833d75d40e8f"
format_stamp: "Formatted at 2026-09-26 05:53:53 on dist-test-slave-7fr8"
server_key: "df9ebf712bbf2771af463d15da00c27a"
server_key_iv: "984392daca041992a1db24d94e165461"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:53.872514 5448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "fe8ceab0a7e0482d9544833d75d40e8f"
format_stamp: "Formatted at 2026-09-26 05:53:53 on dist-test-slave-7fr8"
server_key: "df9ebf712bbf2771af463d15da00c27a"
server_key_iv: "984392daca041992a1db24d94e165461"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:53.877594 5448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 05:53:53.891065 5463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:53.891431 5448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:53:53.891589 5448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "fe8ceab0a7e0482d9544833d75d40e8f"
format_stamp: "Formatted at 2026-09-26 05:53:53 on dist-test-slave-7fr8"
server_key: "df9ebf712bbf2771af463d15da00c27a"
server_key_iv: "984392daca041992a1db24d94e165461"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:53.891708 5448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:53.929113 5448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:53.932471 5448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:53.932703 5448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:53.933030 5448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:53:53.933522 5448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:53:53.933621 5448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:53.933724 5448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:53:53.933801 5448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:53.940336 5448 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:33853
I20260926 05:53:53.941226 5448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:53:53.942616 5578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:33853 every 8 connection(s)
I20260926 05:53:53.947480 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 5448
I20260926 05:53:53.947574 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:53:53.947783 25596 external_mini_cluster.cc:1428] Setting key f5b4955b01950d5b856c173ff02ae850
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Sep 26 05:53:53 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402033, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:53:53.981308 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.255.20:40209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34717
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:53:53.982472 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:53.943300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:53915 (local address 127.24.255.62:34717)
0926 05:53:53.952287 (+ 8987us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:53.952289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:53.952298 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:53:53.952326 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:53.952327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:53.952341 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:53.952400 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:53.953263 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:53.954051 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:53.975312 (+ 21261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:53.975645 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:53.975699 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:53.978548 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:53.978550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:53.978550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:53.978559 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:53.980317 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:53.980981 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:53.980983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:53.980983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:53.981056 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:53.981374 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:53.981376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:53.981376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:53.981540 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:53.981596 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:53.982250 (+ 654us) server_negotiation.cc:299] Negotiation successful
0926 05:53:53.982356 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8950,"thread_start_us":92,"threads_started":1}
I20260926 05:53:53.983292 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:53.943304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34717 (local address 127.24.255.1:53915)
0926 05:53:53.951276 (+ 7972us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:53.951356 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:53.952065 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:53.952569 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:53.952574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:53.952844 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:53.953150 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:53.953156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:53.973729 (+ 20573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:53.973731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:53.975149 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:53.975156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:53.975830 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:53.975831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:53.975997 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:53.976577 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:53.976588 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:53.978400 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:53.980434 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:53.980445 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:53.980448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:53.980881 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:53.981127 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:53.981128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:53.981128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:53.981293 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:53.981833 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:53.981842 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:53.982041 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 05:53:53.982839 (+ 798us) client_negotiation.cc:240] Negotiation successful
0926 05:53:53.983039 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7905,"thread_start_us":81,"threads_started":1}
I20260926 05:53:53.984113 5579 heartbeater.cc:347] Connected to a master server at 127.24.255.62:34717
I20260926 05:53:53.984212 5579 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:53.984414 5579 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
I20260926 05:53:53.984916 5367 ts_manager.cc:194] Registered new tserver with Master: fe8ceab0a7e0482d9544833d75d40e8f (127.24.255.1:33853)
I20260926 05:53:53.985780 5367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:53915
W20260926 05:53:54.117034 5585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:54.117511 5585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:54.117596 5585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:54.119590 5585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:54.119756 5585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:54.119853 5585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:53:54.122965 5585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.255.20:40209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:54.123531 5585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:54.123895 5585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:53:54.136251 5593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:54.138628 5595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:54.139189 5592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:54.139972 5585 server_base.cc:1034] running on GCE node
I20260926 05:53:54.140256 5585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:54.140609 5585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:54.141870 5585 hybrid_clock.cc:648] HybridClock initialized: now 1790402034141850 us; error 36 us; skew 500 ppm
Sep 26 05:53:54 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402034, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:54.149798 5585 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:53:54.151242 5585 webserver.cc:460] Webserver started at http://127.24.255.2:36243/ using document root <none> and password file <none>
I20260926 05:53:54.151618 5585 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:54.151760 5585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:53:54.151971 5585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:53:54 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402034, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
Sep 26 05:53:54 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790402034, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:54.173589 5585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "25c0b1c774a3488cac021367d0671e16"
format_stamp: "Formatted at 2026-09-26 05:53:54 on dist-test-slave-7fr8"
server_key: "dc5a74e16e5252a5a97a7ee5eebdd41e"
server_key_iv: "816d844f6b2a4abe72411417a59acc17"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:54.174038 5585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "25c0b1c774a3488cac021367d0671e16"
format_stamp: "Formatted at 2026-09-26 05:53:54 on dist-test-slave-7fr8"
server_key: "dc5a74e16e5252a5a97a7ee5eebdd41e"
server_key_iv: "816d844f6b2a4abe72411417a59acc17"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:54.178774 5585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260926 05:53:54.200700 5602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:54.200887 5585 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 05:53:54.200959 5585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "25c0b1c774a3488cac021367d0671e16"
format_stamp: "Formatted at 2026-09-26 05:53:54 on dist-test-slave-7fr8"
server_key: "dc5a74e16e5252a5a97a7ee5eebdd41e"
server_key_iv: "816d844f6b2a4abe72411417a59acc17"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:54.201057 5585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:54.264014 5585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:54.266397 5585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:54.266543 5585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:54.266758 5585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:53:54.267107 5585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:53:54.267153 5585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:54.267181 5585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:53:54.267201 5585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:54.273186 5585 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:38035
I20260926 05:53:54.273619 5585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:53:54.275233 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 5585
I20260926 05:53:54.275327 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:53:54.275530 25596 external_mini_cluster.cc:1428] Setting key f6705ecb4478788f835054cfc497fe34
I20260926 05:53:54.276286 5715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:38035 every 8 connection(s)
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
Sep 26 05:53:54 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402034, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
I20260926 05:53:54.296676 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:54.277203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:46377 (local address 127.24.255.62:34717)
0926 05:53:54.278042 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:54.278045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:54.279680 (+ 1635us) server_negotiation.cc:407] Connection header received
0926 05:53:54.280408 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:54.280410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:54.280432 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:54.280488 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:54.281594 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:54.282536 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:54.288242 (+ 5706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:54.288568 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:54.288625 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:54.292887 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:54.292889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:54.292890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:54.292901 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:54.294745 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:54.295385 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:54.295387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:54.295387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:54.295476 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:54.295851 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:54.295853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:54.295853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:54.296016 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:54.296066 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:54.296456 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 05:53:54.296560 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804}
I20260926 05:53:54.297520 5719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:54.276841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34717 (local address 127.24.255.2:46377)
0926 05:53:54.279260 (+ 2419us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:54.279540 (+ 280us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:54.280265 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:54.280720 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:54.280726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:54.281019 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:54.281334 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:54.281341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:54.282686 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:54.282687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:54.284059 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:54.284063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:54.288817 (+ 4754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:54.288819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:54.289044 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:54.290307 (+ 1263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:54.290320 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:54.292720 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:54.294886 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:54.294901 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:54.294904 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:54.295262 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:54.295564 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:54.295565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:54.295565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:54.295743 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:54.296102 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:54.296109 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:54.296298 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 05:53:54.297085 (+ 787us) client_negotiation.cc:240] Negotiation successful
0926 05:53:54.297279 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2343,"thread_start_us":88,"threads_started":1}
I20260926 05:53:54.298286 5716 heartbeater.cc:347] Connected to a master server at 127.24.255.62:34717
I20260926 05:53:54.298393 5716 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:54.298615 5716 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
I20260926 05:53:54.299000 5367 ts_manager.cc:194] Registered new tserver with Master: 25c0b1c774a3488cac021367d0671e16 (127.24.255.2:38035)
I20260926 05:53:54.299441 5367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:46377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:53:54.321863 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.255.20:40209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34717
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:53:54.451541 5722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:54.451855 5722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:54.451897 5722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:54.453918 5722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:54.454010 5722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:54.454057 5722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:53:54.456703 5722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.255.20:40209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:34717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:54.457002 5722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:54.457309 5722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:53:54.467300 5730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:54.468693 5727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:54.468839 5722 server_base.cc:1034] running on GCE node
W20260926 05:53:54.470254 5728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:54.470542 5722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:54.470839 5722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:54.474202 5722 hybrid_clock.cc:648] HybridClock initialized: now 1790402034473995 us; error 192 us; skew 500 ppm
Sep 26 05:53:54 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402034, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:54.477993 5722 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:53:54.478631 5722 webserver.cc:460] Webserver started at http://127.24.255.3:33475/ using document root <none> and password file <none>
I20260926 05:53:54.478960 5722 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:54.479038 5722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:53:54.479187 5722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:53:54 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402034, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:54.500241 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "93b9105e5ea944cab1307f51b9a2da92"
format_stamp: "Formatted at 2026-09-26 05:53:54 on dist-test-slave-7fr8"
server_key: "2f424d8bc8feda27c685aaacbc3a1565"
server_key_iv: "8378fe3135c5999fd14b349baf164861"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:54.500628 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "93b9105e5ea944cab1307f51b9a2da92"
format_stamp: "Formatted at 2026-09-26 05:53:54 on dist-test-slave-7fr8"
server_key: "2f424d8bc8feda27c685aaacbc3a1565"
server_key_iv: "8378fe3135c5999fd14b349baf164861"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:54.502786 5722 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 05:53:54.516286 5737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:54.516449 5722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:53:54.516537 5722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "93b9105e5ea944cab1307f51b9a2da92"
format_stamp: "Formatted at 2026-09-26 05:53:54 on dist-test-slave-7fr8"
server_key: "2f424d8bc8feda27c685aaacbc3a1565"
server_key_iv: "8378fe3135c5999fd14b349baf164861"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:54.516626 5722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:54.529613 5722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:54.531561 5722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:54.531695 5722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:54.531984 5722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:53:54.532366 5722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:53:54.532404 5722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:54.532435 5722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:53:54.532457 5722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:54.540875 5722 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:40135
I20260926 05:53:54.541383 5722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:53:54.542033 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 5722
I20260926 05:53:54.542151 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:53:54.542347 25596 external_mini_cluster.cc:1428] Setting key 056867a1e2d4f00decaf808696103f4f
I20260926 05:53:54.550469 5850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:40135 every 8 connection(s)
Sep 26 05:53:54 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402034, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:53:54.566471 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:54.549319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:50937 (local address 127.24.255.62:34717)
0926 05:53:54.549367 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:54.549369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:54.551950 (+ 2581us) server_negotiation.cc:407] Connection header received
0926 05:53:54.552685 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:54.552687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:54.552699 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:54.552731 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:54.553630 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:54.554446 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:54.556072 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:54.556351 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:54.556397 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:54.562879 (+ 6482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:54.562881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:54.562882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:54.562891 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:54.564617 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:54.565220 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:54.565222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:54.565222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:54.565287 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:54.565599 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:54.565600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:54.565601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:54.565754 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:54.565799 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:54.566270 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 05:53:54.566364 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:53:54.567374 5853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:54.549326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34717 (local address 127.24.255.3:50937)
0926 05:53:54.551770 (+ 2444us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:54.551832 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:54.552553 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:54.552945 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:54.552950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:54.553216 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:54.553513 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:54.553519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:54.554569 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:54.554570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:54.555970 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:54.555973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:54.556509 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:54.556510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:54.556659 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:54.560456 (+ 3797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:54.560470 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:54.562707 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:54.564734 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:54.564748 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:54.564751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:54.565119 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:54.565362 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:54.565363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:54.565363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:54.565519 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:54.565846 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:54.565854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:54.566068 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 05:53:54.566936 (+ 868us) client_negotiation.cc:240] Negotiation successful
0926 05:53:54.567126 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2384,"thread_start_us":117,"threads_started":1}
I20260926 05:53:54.568984 5851 heartbeater.cc:347] Connected to a master server at 127.24.255.62:34717
I20260926 05:53:54.569223 5851 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:54.569494 5851 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
I20260926 05:53:54.570878 5367 ts_manager.cc:194] Registered new tserver with Master: 93b9105e5ea944cab1307f51b9a2da92 (127.24.255.3:40135)
I20260926 05:53:54.571374 5367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:50937
I20260926 05:53:54.581456 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:53:54.582309 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 5448
I20260926 05:53:54.588701 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 5585
I20260926 05:53:54.593274 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 5722
I20260926 05:53:54.598664 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 5335
I20260926 05:53:54.603731 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:34717
--webserver_interface=127.24.255.62
--webserver_port=43171
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:34717
--ranger_config_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.255.20:40209/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.255.20:40209/kms with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:53:54.687160 5854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:54.687541 5854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:54.687568 5854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:54.688923 5854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:53:54.688982 5854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:54.688995 5854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:54.689007 5854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:53:54.689019 5854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:53:54.690855 5854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.255.20:40209/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:34717
--ranger_config_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:34717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=43171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:54.691115 5854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:54.691459 5854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:53:54.696286 5859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:54.696300 5860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:54.696403 5862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:54.696662 5854 server_base.cc:1034] running on GCE node
I20260926 05:53:54.696913 5854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:54.697476 5854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:54.698707 5854 hybrid_clock.cc:648] HybridClock initialized: now 1790402034698652 us; error 70 us; skew 500 ppm
Sep 26 05:53:54 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402034, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:54.701294 5854 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:53:54.701845 5854 webserver.cc:460] Webserver started at http://127.24.255.62:43171/ using document root <none> and password file <none>
I20260926 05:53:54.702069 5854 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:54.702159 5854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 05:53:54 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402034, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:54.748008 5854 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:53:54.748850 5870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:54.749053 5854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:53:54.749157 5854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "a1fe3693d11d4625a95fd2269aa4f45d"
format_stamp: "Formatted at 2026-09-26 05:53:52 on dist-test-slave-7fr8"
server_key: "77b412d331e0ac4b55eb37fdb2a4f086"
server_key_iv: "fc7a4c6c5e8abd674e8c695d85e2d966"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:54.749539 5854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:54.760327 5854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:54.762332 5854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:54.762467 5854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:54.766965 5854 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:34717
I20260926 05:53:54.767021 5925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:34717 every 8 connection(s)
I20260926 05:53:54.767406 5854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:53:54.768962 5926 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:53:54.769824 5926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d: Bootstrap starting.
I20260926 05:53:54.770339 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 5854
I20260926 05:53:54.770663 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.255.20:40209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:33853
--local_ip_for_outbound_sockets=127.24.255.1
--tserver_master_addrs=127.24.255.62:34717
--webserver_port=40565
--webserver_interface=127.24.255.1
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.255.20:40209/kms with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:53:54.779685 5926 log.cc:824] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:53:54.782202 5926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:53:54.782523 5926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d: Bootstrap complete.
I20260926 05:53:54.786634 5926 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34717 } }
I20260926 05:53:54.786989 5926 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fe3693d11d4625a95fd2269aa4f45d, State: Initialized, Role: FOLLOWER
I20260926 05:53:54.787210 5926 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34717 } }
I20260926 05:53:54.787292 5926 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:53:54.787336 5926 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:53:54.787374 5926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:53:54.788853 5926 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34717 } }
I20260926 05:53:54.788981 5926 leader_election.cc:302] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1fe3693d11d4625a95fd2269aa4f45d; no voters:
I20260926 05:53:54.789199 5926 leader_election.cc:288] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 05:53:54.789306 5930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:53:54.789466 5926 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:53:54.789564 5930 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [term 2 LEADER]: Becoming Leader. State: Replica: a1fe3693d11d4625a95fd2269aa4f45d, State: Running, Role: LEADER
I20260926 05:53:54.789702 5930 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34717 } }
I20260926 05:53:54.790002 5930 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34717 } } }
I20260926 05:53:54.790184 5930 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [sys.catalog]: This master's current role is: LEADER
I20260926 05:53:54.790342 5930 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1fe3693d11d4625a95fd2269aa4f45d. Latest consensus state: current_term: 2 leader_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fe3693d11d4625a95fd2269aa4f45d" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 34717 } } }
I20260926 05:53:54.790446 5930 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d [sys.catalog]: This master's current role is: LEADER
I20260926 05:53:54.790583 5940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:53:54.790880 5926 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 05:53:54.791144 5940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:53:54.791326 5940 catalog_manager.cc:1259] Loaded cluster ID: c49b4d6d372047aeb2d87a14ff84619f
I20260926 05:53:54.791394 5940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:53:54.795223 5940 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:53:54.795502 5940 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d: Loaded TSK: 0
I20260926 05:53:54.795724 5940 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 05:53:54.882768 5928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:54.883270 5928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:54.883311 5928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:54.885246 5928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:54.885308 5928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:54.885349 5928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:53:54.888051 5928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.255.20:40209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:33853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=40565
--tserver_master_addrs=127.24.255.62:34717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:54.888388 5928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:54.888685 5928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:53:54.895581 5967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:54.896065 5966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:54.896595 5928 server_base.cc:1034] running on GCE node
W20260926 05:53:54.896816 5969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:54.902297 5928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:54.902765 5928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:54.904029 5928 hybrid_clock.cc:648] HybridClock initialized: now 1790402034904010 us; error 46 us; skew 500 ppm
Sep 26 05:53:54 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402034, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:54.908507 5928 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:53:54.909267 5928 webserver.cc:460] Webserver started at http://127.24.255.1:40565/ using document root <none> and password file <none>
I20260926 05:53:54.909613 5928 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:54.909759 5928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 05:53:54 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402034, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:54.930806 5928 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260926 05:53:54.932000 5977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:54.932242 5928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:53:54.932339 5928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "fe8ceab0a7e0482d9544833d75d40e8f"
format_stamp: "Formatted at 2026-09-26 05:53:53 on dist-test-slave-7fr8"
server_key: "df9ebf712bbf2771af463d15da00c27a"
server_key_iv: "984392daca041992a1db24d94e165461"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:54.932855 5928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:54.971452 5928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:54.973951 5928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:54.974212 5928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:54.974462 5928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:53:54.974852 5928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:53:54.974926 5928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:54.975023 5928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:53:54.975090 5928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:54.980966 5928 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:33853
I20260926 05:53:54.981487 5928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:53:54.984963 6095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:33853 every 8 connection(s)
I20260926 05:53:54.988289 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 5928
I20260926 05:53:54.988595 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.255.20:40209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:38035
--local_ip_for_outbound_sockets=127.24.255.2
--tserver_master_addrs=127.24.255.62:34717
--webserver_port=36243
--webserver_interface=127.24.255.2
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.255.20:40209/kms with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:53:54 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402034, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:53:54.998626 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:54.985812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34717 (local address 127.24.255.1:46043)
0926 05:53:54.986009 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:54.986054 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:54.986748 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:54.987467 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:54.987472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:54.987735 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:54.988025 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:54.988032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:54.990025 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:54.990027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:54.991328 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:54.991332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:54.992170 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:54.992171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:54.992308 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:54.992930 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:54.992939 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:54.994551 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:54.996808 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:54.996816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:54.996818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:54.997138 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:54.997403 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:54.997404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:54.997404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:54.997525 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:54.997958 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:54.997964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:54.998143 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 05:53:54.998230 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 05:53:54.998389 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":123,"threads_started":1}
I20260926 05:53:54.999468 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:54.985878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:46043 (local address 127.24.255.62:34717)
0926 05:53:54.986979 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:54.986989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:54.987033 (+ 44us) server_negotiation.cc:407] Connection header received
0926 05:53:54.987200 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:54.987207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:54.987262 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:54.988187 (+ 925us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:54.988275 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:54.989885 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:54.991447 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:54.992049 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:54.992616 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:54.994672 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:54.994677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:54.994677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:54.994689 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:54.996698 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:54.997231 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:54.997234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:54.997234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:54.997336 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:54.997600 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:54.997601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:54.997601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:54.997885 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:54.999097 (+ 1212us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:54.999171 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 05:53:54.999292 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":272,"threads_started":1}
I20260926 05:53:55.000344 6096 heartbeater.cc:347] Connected to a master server at 127.24.255.62:34717
I20260926 05:53:55.000454 6096 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:55.000661 6096 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
W20260926 05:53:55.001500 6096 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:34717 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 05:53:55.001580 6096 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:55.001642 6096 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
I20260926 05:53:55.001950 6096 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:55.002027 6096 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
W20260926 05:53:55.002357 6096 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:53:55.099138 6101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:55.099442 6101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:55.099478 6101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:55.101217 6101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:55.101281 6101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:55.101323 6101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:53:55.103376 6101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.255.20:40209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:38035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=36243
--tserver_master_addrs=127.24.255.62:34717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:55.103650 6101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:55.103873 6101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:53:55.109182 6107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:55.109191 6109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:55.109380 6106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:55.109633 6101 server_base.cc:1034] running on GCE node
I20260926 05:53:55.109853 6101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:55.110090 6101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:55.111289 6101 hybrid_clock.cc:648] HybridClock initialized: now 1790402035111273 us; error 24 us; skew 500 ppm
Sep 26 05:53:55 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402035, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:55.113924 6101 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:53:55.114459 6101 webserver.cc:460] Webserver started at http://127.24.255.2:36243/ using document root <none> and password file <none>
I20260926 05:53:55.114682 6101 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:55.114744 6101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 05:53:55 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402035, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:55.146693 6101 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.000s sys 0.001s
I20260926 05:53:55.149382 6117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:55.149631 6101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 05:53:55.149736 6101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "25c0b1c774a3488cac021367d0671e16"
format_stamp: "Formatted at 2026-09-26 05:53:54 on dist-test-slave-7fr8"
server_key: "dc5a74e16e5252a5a97a7ee5eebdd41e"
server_key_iv: "816d844f6b2a4abe72411417a59acc17"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:55.150279 6101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:55.174482 6101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:55.177845 6101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:55.177981 6101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:55.178241 6101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:53:55.178563 6101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:53:55.178596 6101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:55.178617 6101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:53:55.178639 6101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:55.186335 6101 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:38035
I20260926 05:53:55.186862 6101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:53:55.187621 6231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:38035 every 8 connection(s)
Sep 26 05:53:55 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402035, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:53:55.196260 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 6101
I20260926 05:53:55.196555 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.255.20:40209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:40135
--local_ip_for_outbound_sockets=127.24.255.3
--tserver_master_addrs=127.24.255.62:34717
--webserver_port=33475
--webserver_interface=127.24.255.3
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.255.20:40209/kms with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:53:55.199811 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:55.188402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:54919 (local address 127.24.255.62:34717)
0926 05:53:55.188452 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:55.188454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:55.188801 (+ 347us) server_negotiation.cc:407] Connection header received
0926 05:53:55.189447 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:55.189448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:55.189462 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:55.189545 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:55.190567 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:55.191403 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:55.192969 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:55.193291 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:55.193365 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:55.196106 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:55.196108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:55.196108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:55.196117 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:55.197864 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:55.198622 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:55.198624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:55.198624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:55.198691 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:55.199012 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:55.199013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:55.199014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:55.199175 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:55.199250 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:55.199627 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 05:53:55.199720 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:53:55.200378 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:55.188394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34717 (local address 127.24.255.2:54919)
0926 05:53:55.188652 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:55.188694 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:55.189328 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:55.189634 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:55.189639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:55.190063 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:55.190445 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:55.190453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:55.191568 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:55.191570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:55.192868 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:55.192872 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:55.193402 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:55.193403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:55.193558 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:55.194262 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:55.194273 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:55.195976 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:55.198187 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:55.198197 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:55.198200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:55.198534 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:55.198773 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:55.198774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:55.198774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:55.198916 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:55.199245 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:55.199251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:55.199461 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 05:53:55.199960 (+ 499us) client_negotiation.cc:240] Negotiation successful
0926 05:53:55.200142 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
I20260926 05:53:55.200882 6232 heartbeater.cc:347] Connected to a master server at 127.24.255.62:34717
I20260926 05:53:55.201004 6232 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:55.201238 6232 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
W20260926 05:53:55.202054 6232 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:34717 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 05:53:55.202169 6232 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:55.202253 6232 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
I20260926 05:53:55.202627 6232 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:55.202713 6232 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
W20260926 05:53:55.203032 6232 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:53:55.301146 6235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:55.301384 6235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:55.301429 6235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:55.302897 6235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:55.302958 6235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:55.302999 6235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:53:55.304963 6235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.255.20:40209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:40135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=33475
--tserver_master_addrs=127.24.255.62:34717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:55.305249 6235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:55.305490 6235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:53:55.311576 6243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:55.311913 6235 server_base.cc:1034] running on GCE node
W20260926 05:53:55.314895 6241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:55.315660 6240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:55.316212 6235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:55.316602 6235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:55.317862 6235 hybrid_clock.cc:648] HybridClock initialized: now 1790402035317828 us; error 41 us; skew 500 ppm
Sep 26 05:53:55 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402035, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:55.320781 6235 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:53:55.321300 6235 webserver.cc:460] Webserver started at http://127.24.255.3:33475/ using document root <none> and password file <none>
I20260926 05:53:55.321542 6235 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:55.321606 6235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 05:53:55 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402035, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:55.338912 6235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:53:55.339900 6252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:55.340121 6235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:53:55.340214 6235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "93b9105e5ea944cab1307f51b9a2da92"
format_stamp: "Formatted at 2026-09-26 05:53:54 on dist-test-slave-7fr8"
server_key: "2f424d8bc8feda27c685aaacbc3a1565"
server_key_iv: "8378fe3135c5999fd14b349baf164861"
server_key_version: "kuduclusterkey@0"
I20260926 05:53:55.340592 6235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:55.361909 6235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:55.363849 6235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:55.363991 6235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:55.364208 6235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:53:55.364557 6235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:53:55.364598 6235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:55.364626 6235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:53:55.364650 6235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:55.371486 6235 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:40135
I20260926 05:53:55.371587 6365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:40135 every 8 connection(s)
I20260926 05:53:55.371958 6235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:53:55.373965 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 6235
Sep 26 05:53:55 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402035, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:53:55.383673 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:55.373350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:54481 (local address 127.24.255.62:34717)
0926 05:53:55.373403 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:55.373405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:55.373629 (+ 224us) server_negotiation.cc:407] Connection header received
0926 05:53:55.374619 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:55.374619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:55.374629 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:55.374668 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:55.375205 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:55.375941 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:55.377423 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:55.377635 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:55.377670 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:55.380254 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:55.380256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:55.380256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:55.380264 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:55.381638 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:55.382198 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:55.382199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:55.382200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:55.382263 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:55.382655 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:55.382655 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:55.382656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:55.382792 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:55.382857 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:55.383477 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 05:53:55.383575 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:53:55.383852 6368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:55.373369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:34717 (local address 127.24.255.3:54481)
0926 05:53:55.373513 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:55.373556 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:55.374241 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:55.374744 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:55.374748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:55.374919 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:55.375119 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:55.375124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:55.376052 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:55.376053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:55.377313 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:55.377317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:55.377781 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:55.377782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:55.377917 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:55.378562 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:55.378572 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:55.380160 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:55.381743 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:55.381752 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:55.381756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:55.382126 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:55.382334 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:55.382334 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:55.382334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:55.382556 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:55.382962 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:55.382981 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:55.383304 (+ 323us) client_negotiation.cc:769] Sending connection context
0926 05:53:55.383433 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 05:53:55.383610 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260926 05:53:55.384348 6366 heartbeater.cc:347] Connected to a master server at 127.24.255.62:34717
I20260926 05:53:55.384459 6366 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:55.384652 6366 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
W20260926 05:53:55.385329 6366 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:34717 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 05:53:55.385394 6366 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:55.385452 6366 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
I20260926 05:53:55.385815 6366 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:55.385890 6366 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
W20260926 05:53:55.386196 6366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:53:55.388460 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:55.374450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.24.255.62:34717)
0926 05:53:55.374551 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:55.374553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:55.378152 (+ 3599us) server_negotiation.cc:407] Connection header received
0926 05:53:55.378182 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:55.378183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:55.378199 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:55.378285 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:55.380475 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:55.381216 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:55.382653 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:55.382943 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:55.383000 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:55.385112 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:55.385114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:55.385114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:55.385121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:55.387120 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:55.387524 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:55.387525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:55.387526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:55.387591 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:55.387832 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:55.387833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:55.387833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:55.387982 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:55.388174 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:55.388259 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 05:53:55.388349 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
Sep 26 05:53:55 dist-test-slave-7fr8 krb5kdc[4891](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790402035, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:56.002532 6096 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:56.002707 6096 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
I20260926 05:53:56.003104 5888 ts_manager.cc:194] Registered new tserver with Master: fe8ceab0a7e0482d9544833d75d40e8f (127.24.255.1:33853)
I20260926 05:53:56.003834 5888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:46043
I20260926 05:53:56.203243 6232 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:56.203411 6232 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
I20260926 05:53:56.203745 5888 ts_manager.cc:194] Registered new tserver with Master: 25c0b1c774a3488cac021367d0671e16 (127.24.255.2:38035)
I20260926 05:53:56.204123 5888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:54919
Sep 26 05:53:56 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402035, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
I20260926 05:53:56.386368 6366 heartbeater.cc:464] Registering TS with master...
I20260926 05:53:56.386552 6366 heartbeater.cc:511] Master 127.24.255.62:34717 requested a full tablet report, sending...
I20260926 05:53:56.386943 5888 ts_manager.cc:194] Registered new tserver with Master: 93b9105e5ea944cab1307f51b9a2da92 (127.24.255.3:40135)
I20260926 05:53:56.387458 5888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:54481
I20260926 05:53:56.390826 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:53:56.400596 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:56.391814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.24.255.62:34717)
0926 05:53:56.392063 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:56.392065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:56.392272 (+ 207us) server_negotiation.cc:407] Connection header received
0926 05:53:56.392411 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:56.392412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:56.392428 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:56.392473 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:56.393201 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.394061 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.395311 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.395639 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.395715 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:56.396896 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:56.396898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:56.396898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:56.396906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:56.399023 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:56.399434 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:56.399436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:56.399436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:56.399499 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:56.399688 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:56.399689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:56.399689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:56.399831 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:56.399921 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:56.400376 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 05:53:56.400469 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":49,"threads_started":1}
I20260926 05:53:56.403204 5888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:53:56.405520 5888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:53:56.421681 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:56.416321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:40135 (local address 127.0.0.1:51964)
0926 05:53:56.416941 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:56.416948 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:56.416987 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:56.417274 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:56.417277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:56.417279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:53:56.417371 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:56.417373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.418686 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.418687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:56.420105 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:56.420108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.421342 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.421343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:56.421434 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:56.421451 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 05:53:56.421521 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 05:53:56.421588 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":4,"thread_start_us":59,"threads_started":1}
I20260926 05:53:56.422180 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:56.416187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.24.255.1:33853)
0926 05:53:56.416518 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:56.416523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:56.417038 (+ 515us) server_negotiation.cc:407] Connection header received
0926 05:53:56.417077 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:56.417079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:56.417095 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:56.417167 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:53:56.417451 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.418599 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.420206 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.421568 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.421624 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:56.421698 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:56.422016 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 05:53:56.422066 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":60,"threads_started":1}
I20260926 05:53:56.421952 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:56.416066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:33853 (local address 127.0.0.1:35240)
0926 05:53:56.416612 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:56.416644 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:56.416725 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:56.417274 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:56.417277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:56.417282 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:53:56.417373 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:56.417374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.419001 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.419002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:56.420105 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:56.420108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.421730 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.421731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:56.421799 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:56.421804 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:53:56.421849 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 05:53:56.421891 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":55,"thread_start_us":97,"threads_started":1}
I20260926 05:53:56.422737 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:56.416420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.24.255.3:40135)
0926 05:53:56.416559 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:56.416562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:56.417038 (+ 476us) server_negotiation.cc:407] Connection header received
0926 05:53:56.417077 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:56.417079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:56.417095 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:56.417178 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:53:56.417470 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.418586 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.420207 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.421159 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.421237 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:56.421307 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:56.422209 (+ 902us) server_negotiation.cc:299] Negotiation successful
0926 05:53:56.422661 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260926 05:53:56.423082 6028 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7bed4739a044d3936ab0aa4cd42310 (DEFAULT_TABLE table=test-table [id=b1605700a28c4b52b626de8eee197a0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:53:56.423486 6028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7bed4739a044d3936ab0aa4cd42310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:53:56.423576 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7bed4739a044d3936ab0aa4cd42310 (DEFAULT_TABLE table=test-table [id=b1605700a28c4b52b626de8eee197a0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:53:56.423888 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7bed4739a044d3936ab0aa4cd42310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:53:56.427167 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:56.416066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:38035 (local address 127.0.0.1:54426)
0926 05:53:56.416827 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:56.416835 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:56.416876 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:56.423305 (+ 6429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:56.423306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:56.423310 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:53:56.423358 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:56.423361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.424586 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.424588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:56.425670 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:56.425674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.426881 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.426882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:56.426954 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:56.426960 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:53:56.426996 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 05:53:56.427044 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260926 05:53:56.427598 6396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:56.416333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.24.255.2:38035)
0926 05:53:56.423124 (+ 6791us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:56.423129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:56.423141 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:53:56.423177 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:56.423178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:56.423194 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:56.423427 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:53:56.423515 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.424479 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.425783 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.426761 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.427370 (+ 609us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:56.427448 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:56.427476 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 05:53:56.427524 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6751,"thread_start_us":50,"threads_started":1}
I20260926 05:53:56.428346 6400 tablet_bootstrap.cc:492] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f: Bootstrap starting.
I20260926 05:53:56.428265 6166 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7bed4739a044d3936ab0aa4cd42310 (DEFAULT_TABLE table=test-table [id=b1605700a28c4b52b626de8eee197a0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:53:56.428601 6166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7bed4739a044d3936ab0aa4cd42310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:53:56.428941 6400 tablet_bootstrap.cc:654] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f: Neither blocks nor log segments found. Creating new log.
I20260926 05:53:56.429204 6400 log.cc:824] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f: Log is configured to *not* fsync() on all Append() calls
I20260926 05:53:56.429960 6400 tablet_bootstrap.cc:492] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f: No bootstrap required, opened a new log
I20260926 05:53:56.430119 6400 ts_tablet_manager.cc:1397] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:53:56.432124 6402 tablet_bootstrap.cc:492] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16: Bootstrap starting.
I20260926 05:53:56.432828 6402 tablet_bootstrap.cc:654] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16: Neither blocks nor log segments found. Creating new log.
I20260926 05:53:56.433187 6402 log.cc:824] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16: Log is configured to *not* fsync() on all Append() calls
I20260926 05:53:56.433176 6400 raft_consensus.cc:348] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c0b1c774a3488cac021367d0671e16" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 38035 } } peers { permanent_uuid: "fe8ceab0a7e0482d9544833d75d40e8f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 33853 } } peers { permanent_uuid: "93b9105e5ea944cab1307f51b9a2da92" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 40135 } }
I20260926 05:53:56.433391 6400 raft_consensus.cc:374] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:53:56.433459 6400 raft_consensus.cc:729] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8ceab0a7e0482d9544833d75d40e8f, State: Initialized, Role: FOLLOWER
I20260926 05:53:56.433614 6400 consensus_queue.cc:260] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c0b1c774a3488cac021367d0671e16" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 38035 } } peers { permanent_uuid: "fe8ceab0a7e0482d9544833d75d40e8f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 33853 } } peers { permanent_uuid: "93b9105e5ea944cab1307f51b9a2da92" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 40135 } }
I20260926 05:53:56.433975 6400 ts_tablet_manager.cc:1428] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 05:53:56.434158 6402 tablet_bootstrap.cc:492] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16: No bootstrap required, opened a new log
I20260926 05:53:56.434202 6096 heartbeater.cc:503] Master 127.24.255.62:34717 was elected leader, sending a full tablet report...
I20260926 05:53:56.434231 6402 ts_tablet_manager.cc:1397] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:53:56.437196 6402 raft_consensus.cc:348] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c0b1c774a3488cac021367d0671e16" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 38035 } } peers { permanent_uuid: "fe8ceab0a7e0482d9544833d75d40e8f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 33853 } } peers { permanent_uuid: "93b9105e5ea944cab1307f51b9a2da92" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 40135 } }
I20260926 05:53:56.437397 6402 raft_consensus.cc:374] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:53:56.437426 6402 raft_consensus.cc:729] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25c0b1c774a3488cac021367d0671e16, State: Initialized, Role: FOLLOWER
I20260926 05:53:56.437664 6402 consensus_queue.cc:260] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c0b1c774a3488cac021367d0671e16" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 38035 } } peers { permanent_uuid: "fe8ceab0a7e0482d9544833d75d40e8f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 33853 } } peers { permanent_uuid: "93b9105e5ea944cab1307f51b9a2da92" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 40135 } }
I20260926 05:53:56.438012 6232 heartbeater.cc:503] Master 127.24.255.62:34717 was elected leader, sending a full tablet report...
I20260926 05:53:56.438310 6399 tablet_bootstrap.cc:492] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92: Bootstrap starting.
W20260926 05:53:56.438503 6233 tablet.cc:2369] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:53:56.437954 6402 ts_tablet_manager.cc:1428] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 05:53:56.439105 6399 tablet_bootstrap.cc:654] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92: Neither blocks nor log segments found. Creating new log.
I20260926 05:53:56.439481 6399 log.cc:824] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92: Log is configured to *not* fsync() on all Append() calls
I20260926 05:53:56.440474 6399 tablet_bootstrap.cc:492] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92: No bootstrap required, opened a new log
I20260926 05:53:56.440548 6399 ts_tablet_manager.cc:1397] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:53:56.443557 6399 raft_consensus.cc:348] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c0b1c774a3488cac021367d0671e16" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 38035 } } peers { permanent_uuid: "fe8ceab0a7e0482d9544833d75d40e8f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 33853 } } peers { permanent_uuid: "93b9105e5ea944cab1307f51b9a2da92" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 40135 } }
I20260926 05:53:56.443853 6399 raft_consensus.cc:374] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:53:56.443940 6399 raft_consensus.cc:729] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b9105e5ea944cab1307f51b9a2da92, State: Initialized, Role: FOLLOWER
I20260926 05:53:56.444095 6399 consensus_queue.cc:260] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c0b1c774a3488cac021367d0671e16" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 38035 } } peers { permanent_uuid: "fe8ceab0a7e0482d9544833d75d40e8f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 33853 } } peers { permanent_uuid: "93b9105e5ea944cab1307f51b9a2da92" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 40135 } }
I20260926 05:53:56.444443 6399 ts_tablet_manager.cc:1428] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 05:53:56.444617 6366 heartbeater.cc:503] Master 127.24.255.62:34717 was elected leader, sending a full tablet report...
W20260926 05:53:56.486784 6097 tablet.cc:2369] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 05:53:56 dist-test-slave-7fr8 krb5kdc[4891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402035, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for HTTP/127.24.255.20@KRBTEST.COM
W20260926 05:53:56.624048 6367 tablet.cc:2369] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:53:56.684145 6407 raft_consensus.cc:482] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:53:56.684309 6407 raft_consensus.cc:504] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c0b1c774a3488cac021367d0671e16" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 38035 } } peers { permanent_uuid: "fe8ceab0a7e0482d9544833d75d40e8f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 33853 } } peers { permanent_uuid: "93b9105e5ea944cab1307f51b9a2da92" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 40135 } }
I20260926 05:53:56.684734 6407 leader_election.cc:288] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25c0b1c774a3488cac021367d0671e16 (127.24.255.2:38035), fe8ceab0a7e0482d9544833d75d40e8f (127.24.255.1:33853)
I20260926 05:53:56.688760 6396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:56.685227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:51459 (local address 127.24.255.2:38035)
0926 05:53:56.685271 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:56.685273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:56.685284 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:53:56.685393 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:56.685394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:56.685404 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:56.685434 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:53:56.685665 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.686376 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.687733 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.688313 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.688360 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:56.688371 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:56.688607 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 05:53:56.688657 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:53:56.689577 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:56.684878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:38035 (local address 127.24.255.3:51459)
0926 05:53:56.685194 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:56.685216 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:56.685269 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:56.685503 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:56.685504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:56.685506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:53:56.685587 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:56.685590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.686482 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.686483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:56.687623 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:56.687627 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.688451 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.688453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:56.688522 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:56.688526 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:53:56.689365 (+ 839us) client_negotiation.cc:240] Negotiation successful
0926 05:53:56.689470 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":126,"threads_started":1}
I20260926 05:53:56.690143 6186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7bed4739a044d3936ab0aa4cd42310" candidate_uuid: "93b9105e5ea944cab1307f51b9a2da92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25c0b1c774a3488cac021367d0671e16" is_pre_election: true
I20260926 05:53:56.690333 6186 raft_consensus.cc:2454] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93b9105e5ea944cab1307f51b9a2da92 in term 0.
I20260926 05:53:56.691296 6255 leader_election.cc:302] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25c0b1c774a3488cac021367d0671e16, 93b9105e5ea944cab1307f51b9a2da92; no voters:
I20260926 05:53:56.691516 6407 raft_consensus.cc:2781] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:53:56.691576 6407 raft_consensus.cc:482] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:53:56.691599 6407 raft_consensus.cc:3037] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:53:56.692513 6407 raft_consensus.cc:504] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c0b1c774a3488cac021367d0671e16" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 38035 } } peers { permanent_uuid: "fe8ceab0a7e0482d9544833d75d40e8f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 33853 } } peers { permanent_uuid: "93b9105e5ea944cab1307f51b9a2da92" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 40135 } }
I20260926 05:53:56.692663 6407 leader_election.cc:288] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [CANDIDATE]: Term 1 election: Requested vote from peers 25c0b1c774a3488cac021367d0671e16 (127.24.255.2:38035), fe8ceab0a7e0482d9544833d75d40e8f (127.24.255.1:33853)
I20260926 05:53:56.693156 6186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7bed4739a044d3936ab0aa4cd42310" candidate_uuid: "93b9105e5ea944cab1307f51b9a2da92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25c0b1c774a3488cac021367d0671e16"
I20260926 05:53:56.693250 6186 raft_consensus.cc:3037] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:53:56.699044 6186 raft_consensus.cc:2454] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93b9105e5ea944cab1307f51b9a2da92 in term 1.
I20260926 05:53:56.699397 6255 leader_election.cc:302] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25c0b1c774a3488cac021367d0671e16, 93b9105e5ea944cab1307f51b9a2da92; no voters:
I20260926 05:53:56.699553 6407 raft_consensus.cc:2781] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:53:56.699707 6407 raft_consensus.cc:686] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [term 1 LEADER]: Becoming Leader. State: Replica: 93b9105e5ea944cab1307f51b9a2da92, State: Running, Role: LEADER
I20260926 05:53:56.699803 6407 consensus_queue.cc:237] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c0b1c774a3488cac021367d0671e16" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 38035 } } peers { permanent_uuid: "fe8ceab0a7e0482d9544833d75d40e8f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 33853 } } peers { permanent_uuid: "93b9105e5ea944cab1307f51b9a2da92" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 40135 } }
I20260926 05:53:56.700721 5888 catalog_manager.cc:5495] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93b9105e5ea944cab1307f51b9a2da92 (127.24.255.3). New cstate: current_term: 1 leader_uuid: "93b9105e5ea944cab1307f51b9a2da92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25c0b1c774a3488cac021367d0671e16" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 38035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe8ceab0a7e0482d9544833d75d40e8f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 33853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93b9105e5ea944cab1307f51b9a2da92" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 40135 } health_report { overall_health: HEALTHY } } }
I20260926 05:53:56.708212 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:56.685966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:45259 (local address 127.24.255.1:33853)
0926 05:53:56.686019 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:56.686021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:56.690002 (+ 3981us) server_negotiation.cc:407] Connection header received
0926 05:53:56.690124 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:56.690125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:56.690136 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:56.690165 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:53:56.690339 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.691070 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.707238 (+ 16168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.707822 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.707910 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:56.707921 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:56.708062 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 05:53:56.708133 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:53:56.708500 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:56.684926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:33853 (local address 127.24.255.3:45259)
0926 05:53:56.689948 (+ 5022us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:56.689960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:56.690033 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:56.690230 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:56.690231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:56.690233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:53:56.690269 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:56.690271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.705880 (+ 15609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.705881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:56.707067 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:56.707073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.707910 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.707911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:56.707973 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:56.707977 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:53:56.708360 (+ 383us) client_negotiation.cc:240] Negotiation successful
0926 05:53:56.708432 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5008,"thread_start_us":160,"threads_started":1}
I20260926 05:53:56.708962 6049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7bed4739a044d3936ab0aa4cd42310" candidate_uuid: "93b9105e5ea944cab1307f51b9a2da92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8ceab0a7e0482d9544833d75d40e8f" is_pre_election: true
I20260926 05:53:56.709132 6049 raft_consensus.cc:2454] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93b9105e5ea944cab1307f51b9a2da92 in term 0.
I20260926 05:53:56.709282 6048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7bed4739a044d3936ab0aa4cd42310" candidate_uuid: "93b9105e5ea944cab1307f51b9a2da92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8ceab0a7e0482d9544833d75d40e8f"
I20260926 05:53:56.709349 6048 raft_consensus.cc:3037] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:53:56.710062 6048 raft_consensus.cc:2454] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93b9105e5ea944cab1307f51b9a2da92 in term 1.
I20260926 05:53:56.777626 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:56.774272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.24.255.3:40135)
0926 05:53:56.774331 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:56.774332 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:56.774339 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:53:56.774380 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:56.774380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:56.774398 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:56.774465 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:53:56.774636 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.775381 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.776475 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:56.776762 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:56.776823 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:56.777054 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:53:56.777197 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:53:56.777274 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:56.777514 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 05:53:56.777541 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:53:56.779836 6048 raft_consensus.cc:1261] T 3f7bed4739a044d3936ab0aa4cd42310 P fe8ceab0a7e0482d9544833d75d40e8f [term 1 FOLLOWER]: Refusing update from remote peer 93b9105e5ea944cab1307f51b9a2da92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:53:56.780309 6412 consensus_queue.cc:1035] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8ceab0a7e0482d9544833d75d40e8f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:53:56.782676 6186 raft_consensus.cc:1261] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16 [term 1 FOLLOWER]: Refusing update from remote peer 93b9105e5ea944cab1307f51b9a2da92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:53:56.783071 6412 consensus_queue.cc:1035] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25c0b1c774a3488cac021367d0671e16" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 38035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:53:56.787633 6415 mvcc.cc:205] Tried to move back new op lower bound from 7333486742647439360 to 7333486742323040256. Current Snapshot: MvccSnapshot[applied={T|T < 7333486742647439360}]
I20260926 05:53:56.787915 6418 mvcc.cc:205] Tried to move back new op lower bound from 7333486742647439360 to 7333486742323040256. Current Snapshot: MvccSnapshot[applied={T|T < 7333486742647439360}]
I20260926 05:53:56.791429 6416 mvcc.cc:205] Tried to move back new op lower bound from 7333486742647439360 to 7333486742323040256. Current Snapshot: MvccSnapshot[applied={T|T < 7333486742647439360}]
I20260926 05:53:56.793989 5888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:53:56.794202 5888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:53:56.795104 5888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1fe3693d11d4625a95fd2269aa4f45d: Sending DeleteTablet for 3 replicas of tablet 3f7bed4739a044d3936ab0aa4cd42310
I20260926 05:53:56.795696 6166 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f7bed4739a044d3936ab0aa4cd42310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:53:56 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:54426
I20260926 05:53:56.795689 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f7bed4739a044d3936ab0aa4cd42310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:53:56 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:51964
I20260926 05:53:56.795966 6028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f7bed4739a044d3936ab0aa4cd42310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:53:56 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:35240
I20260926 05:53:56.796757 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 5928
I20260926 05:53:56.802306 6424 tablet_replica.cc:330] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92: stopping tablet replica
I20260926 05:53:56.802568 6424 raft_consensus.cc:2229] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:53:56.802845 6424 raft_consensus.cc:2258] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:53:56.803845 6425 tablet_replica.cc:330] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16: stopping tablet replica
I20260926 05:53:56.804085 6425 raft_consensus.cc:2229] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:53:56.804353 6424 ts_tablet_manager.cc:1905] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:53:56.806248 6424 ts_tablet_manager.cc:1918] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:53:56.806408 6424 log.cc:1196] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/3f7bed4739a044d3936ab0aa4cd42310
I20260926 05:53:56.806789 6424 ts_tablet_manager.cc:1939] T 3f7bed4739a044d3936ab0aa4cd42310 P 93b9105e5ea944cab1307f51b9a2da92: Deleting consensus metadata
I20260926 05:53:56.807432 5871 catalog_manager.cc:4841] TS 93b9105e5ea944cab1307f51b9a2da92 (127.24.255.3:40135): tablet 3f7bed4739a044d3936ab0aa4cd42310 (table test-table [id=b1605700a28c4b52b626de8eee197a0a]) successfully deleted
I20260926 05:53:56.808020 6425 raft_consensus.cc:2258] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:53:56.813657 6425 ts_tablet_manager.cc:1905] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 05:53:56.815730 5872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:33853 (error 108)
I20260926 05:53:56.816125 6425 ts_tablet_manager.cc:1918] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:53:56.816212 6425 log.cc:1196] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/3f7bed4739a044d3936ab0aa4cd42310
I20260926 05:53:56.816473 6425 ts_tablet_manager.cc:1939] T 3f7bed4739a044d3936ab0aa4cd42310 P 25c0b1c774a3488cac021367d0671e16: Deleting consensus metadata
I20260926 05:53:56.816969 5873 catalog_manager.cc:4841] TS 25c0b1c774a3488cac021367d0671e16 (127.24.255.2:38035): tablet 3f7bed4739a044d3936ab0aa4cd42310 (table test-table [id=b1605700a28c4b52b626de8eee197a0a]) successfully deleted
I20260926 05:53:56.817444 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:56.817310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:33853 (local address 127.0.0.1:35252)
0926 05:53:56.817358 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:56.817398 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 05:53:56.817457 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 6101
W20260926 05:53:56.819664 5872 catalog_manager.cc:4584] TS fe8ceab0a7e0482d9544833d75d40e8f (127.24.255.1:33853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f7bed4739a044d3936ab0aa4cd42310: Network error: Client connection negotiation failed: client connection to 127.24.255.1:33853: connect: Connection refused (error 111)
I20260926 05:53:56.823982 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 6235
I20260926 05:53:56.829634 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 5854
I20260926 05:53:56.834533 25596 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 05:53:57.169945 25596 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 05:53:57.170050 25596 mini_ranger.cc:66] Stopping Ranger...
I20260926 05:53:57.505462 25596 mini_ranger.cc:68] Stopped Ranger
2026-09-26 05:53:57.505 UTC [4921] LOG: received smart shutdown request
2026-09-26 05:53:57.510 UTC [4921] LOG: background worker "logical replication launcher" (PID 4930) exited with exit code 1
2026-09-26 05:53:57.511 UTC [4925] LOG: shutting down
2026-09-26 05:53:57.712 UTC [4921] LOG: database system is shut down
2026-09-26T05:53:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51815 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:53:57 dist-test-slave-7fr8 krb5kdc[6436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:53:57 dist-test-slave-7fr8 krb5kdc[6436](info): set up 2 sockets
Sep 26 05:53:57 dist-test-slave-7fr8 krb5kdc[6436](info): commencing operation
krb5kdc: starting...
W20260926 05:53:58.782608 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:53:58 dist-test-slave-7fr8 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:53:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:53:58.904663 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:38455
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:38455,127.24.255.61:38439,127.24.255.60:35547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:53:58.996289 6449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:58.996546 6449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:58.996580 6449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:58.997857 6449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:53:58.997911 6449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:58.997931 6449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:58.997946 6449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:53:58.997962 6449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:53:58.999732 6449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:38455,127.24.255.61:38439,127.24.255.60:35547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:38455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:59.000015 6449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:59.000233 6449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:53:59.005333 6449 server_base.cc:1034] running on GCE node
W20260926 05:53:59.005229 6454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:59.005261 6457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:59.005235 6455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:59.005750 6449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:59.006033 6449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:59.007205 6449 hybrid_clock.cc:648] HybridClock initialized: now 1790402039007187 us; error 29 us; skew 500 ppm
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402039, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:59.010324 6449 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:53:59.010774 6449 webserver.cc:460] Webserver started at http://127.24.255.62:38839/ using document root <none> and password file <none>
I20260926 05:53:59.010999 6449 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:59.011045 6449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:53:59.011175 6449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:53:59.012290 6449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "202ead50c74c4e23bf0955dc65c54007"
format_stamp: "Formatted at 2026-09-26 05:53:59 on dist-test-slave-7fr8"
server_key: "b1f23649c98fcb80a2cf1b45ffce1554"
server_key_iv: "1f46deea54a9667c5550e254990aa75a"
server_key_version: "encryptionkey@0"
I20260926 05:53:59.012625 6449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "202ead50c74c4e23bf0955dc65c54007"
format_stamp: "Formatted at 2026-09-26 05:53:59 on dist-test-slave-7fr8"
server_key: "b1f23649c98fcb80a2cf1b45ffce1554"
server_key_iv: "1f46deea54a9667c5550e254990aa75a"
server_key_version: "encryptionkey@0"
I20260926 05:53:59.014206 6449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:53:59.015084 6464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:59.015342 6449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:53:59.015437 6449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "202ead50c74c4e23bf0955dc65c54007"
format_stamp: "Formatted at 2026-09-26 05:53:59 on dist-test-slave-7fr8"
server_key: "b1f23649c98fcb80a2cf1b45ffce1554"
server_key_iv: "1f46deea54a9667c5550e254990aa75a"
server_key_version: "encryptionkey@0"
I20260926 05:53:59.015509 6449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:59.038574 6449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:59.041654 6449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:59.041822 6449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:59.046147 6449 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:38455
I20260926 05:53:59.046173 6516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:38455 every 8 connection(s)
I20260926 05:53:59.046545 6449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:53:59.047421 6517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:53:59.049131 6517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:53:59.051144 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 6449
I20260926 05:53:59.051262 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:53:59.051427 25596 external_mini_cluster.cc:1428] Setting key 9bd81c63e3a5e1aa88e5316fd5e43f7e
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402039, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.255.61@KRBTEST.COM; defaulting to no policy
I20260926 05:53:59.060456 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.050320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38455 (local address 127.0.0.1:34426)
0926 05:53:59.050554 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.050618 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:59.051216 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:59.051628 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:59.051632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:59.051984 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:59.052221 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.052226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.053499 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.053500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.054311 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.054314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.054910 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.054911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.054998 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.055672 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:59.055683 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:59.057317 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:59.059027 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.059033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.059036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.059235 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.059447 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.059447 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.059448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.059541 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.059884 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:59.059888 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:59.059958 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 05:53:59.060042 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 05:53:59.060264 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":76,"thread_start_us":101,"threads_started":1}
I20260926 05:53:59.060454 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.050519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.24.255.62:38455)
0926 05:53:59.050643 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.050646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.050690 (+ 44us) server_negotiation.cc:407] Connection header received
0926 05:53:59.051474 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.051477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.051504 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.051604 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.052542 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.053365 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.054417 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.054813 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.054905 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.057413 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.057421 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.057421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.057436 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.058939 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.059308 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.059311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.059311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.059387 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.059597 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.059597 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.059598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.059822 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.059895 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.060082 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.060191 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":44,"threads_started":1}
I20260926 05:53:59.061342 6517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:53:59.061745 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.061595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38439 (local address 127.0.0.1:57490)
0926 05:53:59.061659 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.061715 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 05:53:59.061914 6465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38439: connect: Connection refused (error 111)
I20260926 05:53:59.062345 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.062262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38439 (local address 127.0.0.1:57498)
0926 05:53:59.062301 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.062325 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 05:53:59.062471 6517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.61:38439: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38439: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.61.keytab.
Entry for principal kudu/127.24.255.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.61.keytab.
I20260926 05:53:59.081019 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.61:38439
--webserver_interface=127.24.255.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.61.keytab
--principal=kudu/127.24.255.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:38455,127.24.255.61:38439,127.24.255.60:35547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:53:59.111726 6517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } attempt: 1
I20260926 05:53:59.112257 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.112135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38439 (local address 127.0.0.1:57514)
0926 05:53:59.112192 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.112228 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 05:53:59.112900 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.112763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38439 (local address 127.0.0.1:57518)
0926 05:53:59.112835 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.112866 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 05:53:59.113001 6517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.61:38439: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38439: connect: Connection refused (error 111)
W20260926 05:53:59.169415 6525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:59.169651 6525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:59.169683 6525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:59.170986 6525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:53:59.171042 6525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:59.171061 6525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:59.171077 6525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:53:59.171092 6525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:53:59.172816 6525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:38455,127.24.255.61:38439,127.24.255.60:35547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.61.keytab
--principal=kudu/127.24.255.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.61:38439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.255.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:59.173118 6525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:59.173345 6525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:53:59.178715 6533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:59.178763 6531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:59.178733 6530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:59.179184 6525 server_base.cc:1034] running on GCE node
I20260926 05:53:59.179477 6525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:59.179796 6525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:59.180996 6525 hybrid_clock.cc:648] HybridClock initialized: now 1790402039180936 us; error 83 us; skew 500 ppm
I20260926 05:53:59.181162 6517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } attempt: 2
I20260926 05:53:59.181725 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.181558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38439 (local address 127.0.0.1:57534)
0926 05:53:59.181633 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.181682 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 05:53:59.182178 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.182023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38439 (local address 127.0.0.1:57540)
0926 05:53:59.182065 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.182146 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 05:53:59.182276 6517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.61:38439: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38439: connect: Connection refused (error 111)
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402039, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:59.189162 6525 init.cc:377] Logged in from keytab as kudu/127.24.255.61@KRBTEST.COM (short username kudu)
I20260926 05:53:59.189649 6525 webserver.cc:460] Webserver started at http://127.24.255.61:45939/ using document root <none> and password file <none>
I20260926 05:53:59.189858 6525 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:59.189905 6525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:53:59.189998 6525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:53:59.191138 6525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/data/instance:
uuid: "e711288ccf75487885965714ae71f6ab"
format_stamp: "Formatted at 2026-09-26 05:53:59 on dist-test-slave-7fr8"
server_key: "bf26bfcb00f2f03eed3dbabe39161d57"
server_key_iv: "d6b44259dc4d71104b0f244983d29818"
server_key_version: "encryptionkey@0"
I20260926 05:53:59.191461 6525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/wal/instance:
uuid: "e711288ccf75487885965714ae71f6ab"
format_stamp: "Formatted at 2026-09-26 05:53:59 on dist-test-slave-7fr8"
server_key: "bf26bfcb00f2f03eed3dbabe39161d57"
server_key_iv: "d6b44259dc4d71104b0f244983d29818"
server_key_version: "encryptionkey@0"
I20260926 05:53:59.192821 6525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:53:59.193812 6540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:59.194278 6525 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:53:59.194393 6525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/wal
uuid: "e711288ccf75487885965714ae71f6ab"
format_stamp: "Formatted at 2026-09-26 05:53:59 on dist-test-slave-7fr8"
server_key: "bf26bfcb00f2f03eed3dbabe39161d57"
server_key_iv: "d6b44259dc4d71104b0f244983d29818"
server_key_version: "encryptionkey@0"
I20260926 05:53:59.194476 6525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:59.218218 6525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:59.220873 6525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:59.221035 6525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:59.226164 6525 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.61:38439
I20260926 05:53:59.226307 6593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.61:38439 every 8 connection(s)
I20260926 05:53:59.226637 6525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/data/info.pb
I20260926 05:53:59.227020 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 6525
I20260926 05:53:59.227123 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-1/wal/instance
I20260926 05:53:59.227347 25596 external_mini_cluster.cc:1428] Setting key 950c95e12ad8da14c7179094133c377d
I20260926 05:53:59.230605 6594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:53:59.233202 6594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.255.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.60@KRBTEST.COM" created.
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402039, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.61@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:53:59.245783 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.234308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.24.255.62:38455)
0926 05:53:59.234363 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.234366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.234975 (+ 609us) server_negotiation.cc:407] Connection header received
0926 05:53:59.235675 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.235677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.235691 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.235765 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.236641 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.237445 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.239355 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.239680 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.239760 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.242409 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.242412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.242412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.242422 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.244036 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.244612 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.244614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.244614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.244701 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.245004 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.245005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.245005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.245160 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.245211 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.245590 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.245689 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:53:59.245968 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.234088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38455 (local address 127.0.0.1:34440)
0926 05:53:59.234813 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.234867 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:59.235530 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:59.235865 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:59.235870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:59.236222 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:59.236526 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.236533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.237582 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.237584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.239044 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.239049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.239846 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.239847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.240022 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.240571 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:59.240583 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:59.242266 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:59.244141 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.244151 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.244155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.244494 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.244783 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.244784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.244785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.244912 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.245244 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:59.245252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:59.245441 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 05:53:59.245535 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 05:53:59.245719 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":84,"threads_started":1}
I20260926 05:53:59.246665 6594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402039, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.61@KRBTEST.COM for kudu/127.24.255.61@KRBTEST.COM
I20260926 05:53:59.256213 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.246931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38439 (local address 127.0.0.1:57550)
0926 05:53:59.246973 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.246983 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:59.247017 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:59.247614 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:59.247614 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:59.247785 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:59.247847 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.247849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.249151 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.249153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.249963 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.249967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.250667 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.250668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.250736 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.251250 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:59.251256 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:59.252707 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:59.254773 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.254774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.254775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.254994 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.255500 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.255500 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.255501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.255576 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.255880 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:59.255881 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:59.255920 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 05:53:59.255989 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 05:53:59.256107 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 05:53:59.256346 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.247268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.24.255.61:38439)
0926 05:53:59.247414 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.247419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.247444 (+ 25us) server_negotiation.cc:407] Connection header received
0926 05:53:59.247465 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.247468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.247481 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.247581 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.247923 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.249041 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.250131 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.250576 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.250651 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.252802 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.252816 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.252816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.252824 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.254691 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.255137 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.255145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.255146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.255395 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.255627 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.255628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.255629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.255803 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.255887 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.256031 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.256216 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260926 05:53:59.257323 6594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:53:59.257833 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.257643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35547 (local address 127.0.0.1:49102)
0926 05:53:59.257680 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.257784 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 05:53:59.258088 6541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
I20260926 05:53:59.258948 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.258714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35547 (local address 127.0.0.1:49118)
0926 05:53:59.258834 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.258873 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260926 05:53:59.259104 6594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:35547: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.60.keytab.
Entry for principal kudu/127.24.255.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.60.keytab.
I20260926 05:53:59.261456 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.60:35547
--webserver_interface=127.24.255.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.60.keytab
--principal=kudu/127.24.255.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:38455,127.24.255.61:38439,127.24.255.60:35547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:53:59.273478 6517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } attempt: 3
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402039, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for kudu/127.24.255.61@KRBTEST.COM
I20260926 05:53:59.282608 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.273867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38439 (local address 127.0.0.1:57554)
0926 05:53:59.273928 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.273940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:59.274123 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:59.274339 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:59.274340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:59.274504 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:59.274587 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.274589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.275684 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.275685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.276284 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.276288 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.276857 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.276860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.276983 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.277635 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:59.277641 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:59.279339 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:59.281302 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.281303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.281303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.281495 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.281765 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.281766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.281767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.281885 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.282270 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:59.282272 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:59.282335 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 05:53:59.282396 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:53:59.282503 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:53:59.282611 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.273937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.24.255.61:38439)
0926 05:53:59.274056 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.274058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.274076 (+ 18us) server_negotiation.cc:407] Connection header received
0926 05:53:59.274220 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.274221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.274235 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.274303 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.274654 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.275585 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.276366 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.276691 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.276762 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.279437 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.279439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.279440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.279449 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.281170 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.281565 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.281566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.281567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.281627 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.281980 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.281981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.281981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.282152 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.282224 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.282430 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.282524 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 05:53:59.283042 6517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:53:59.283412 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.283314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35547 (local address 127.0.0.1:49124)
0926 05:53:59.283359 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.283388 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 05:53:59.283758 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.283642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35547 (local address 127.0.0.1:49140)
0926 05:53:59.283706 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.283736 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 05:53:59.283844 6517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:35547: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
I20260926 05:53:59.308425 6594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } attempt: 1
I20260926 05:53:59.308987 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.308835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35547 (local address 127.0.0.1:49146)
0926 05:53:59.308895 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.308952 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:53:59.309552 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.309447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35547 (local address 127.0.0.1:49158)
0926 05:53:59.309491 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.309524 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:53:59.309665 6594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:35547: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
I20260926 05:53:59.315042 6517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } attempt: 1
I20260926 05:53:59.315563 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.315402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35547 (local address 127.0.0.1:49160)
0926 05:53:59.315483 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.315528 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 05:53:59.315917 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.315798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35547 (local address 127.0.0.1:49164)
0926 05:53:59.315860 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.315892 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 05:53:59.316043 6517 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:35547: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
W20260926 05:53:59.351219 6602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:53:59.351450 6602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:53:59.351472 6602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:53:59.352679 6602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:53:59.352726 6602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:53:59.352741 6602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:53:59.352751 6602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:53:59.352762 6602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:53:59.354398 6602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:38455,127.24.255.61:38439,127.24.255.60:35547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.60.keytab
--principal=kudu/127.24.255.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.60:35547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.255.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:53:59.354632 6602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:53:59.354815 6602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:53:59.359725 6602 server_base.cc:1034] running on GCE node
W20260926 05:53:59.359678 6607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:59.359685 6610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:53:59.359685 6608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:53:59.360186 6602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:53:59.360481 6602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:53:59.361641 6602 hybrid_clock.cc:648] HybridClock initialized: now 1790402039361605 us; error 70 us; skew 500 ppm
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402039, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:53:59.364190 6602 init.cc:377] Logged in from keytab as kudu/127.24.255.60@KRBTEST.COM (short username kudu)
I20260926 05:53:59.364696 6602 webserver.cc:460] Webserver started at http://127.24.255.60:39205/ using document root <none> and password file <none>
I20260926 05:53:59.364902 6602 fs_manager.cc:359] Metadata directory not provided
I20260926 05:53:59.364954 6602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:53:59.365080 6602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:53:59.366338 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/data/instance:
uuid: "91c6146929744a33af9090241b118be8"
format_stamp: "Formatted at 2026-09-26 05:53:59 on dist-test-slave-7fr8"
server_key: "d693a030b584981f18d4e2d0746ab11a"
server_key_iv: "b76ee4e7f797243ec46230189dd33e4d"
server_key_version: "encryptionkey@0"
I20260926 05:53:59.366730 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/wal/instance:
uuid: "91c6146929744a33af9090241b118be8"
format_stamp: "Formatted at 2026-09-26 05:53:59 on dist-test-slave-7fr8"
server_key: "d693a030b584981f18d4e2d0746ab11a"
server_key_iv: "b76ee4e7f797243ec46230189dd33e4d"
server_key_version: "encryptionkey@0"
I20260926 05:53:59.368112 6602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:53:59.368942 6617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:53:59.369122 6602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:53:59.369200 6602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/wal
uuid: "91c6146929744a33af9090241b118be8"
format_stamp: "Formatted at 2026-09-26 05:53:59 on dist-test-slave-7fr8"
server_key: "d693a030b584981f18d4e2d0746ab11a"
server_key_iv: "b76ee4e7f797243ec46230189dd33e4d"
server_key_version: "encryptionkey@0"
I20260926 05:53:59.369275 6602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:53:59.377799 6594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } attempt: 2
I20260926 05:53:59.378320 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.378193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35547 (local address 127.0.0.1:49178)
0926 05:53:59.378246 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.378288 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 05:53:59.378712 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.378643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35547 (local address 127.0.0.1:49194)
0926 05:53:59.378675 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.378697 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 05:53:59.378821 6594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:35547: Network error: Client connection negotiation failed: client connection to 127.24.255.60:35547: connect: Connection refused (error 111)
I20260926 05:53:59.382539 6602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:53:59.385128 6602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:53:59.385257 6602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:53:59.389346 6602 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.60:35547
I20260926 05:53:59.389385 6669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.60:35547 every 8 connection(s)
I20260926 05:53:59.389744 6602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/data/info.pb
I20260926 05:53:59.390661 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:53:59.391191 6517 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } attempt: 2
I20260926 05:53:59.392472 6670 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402039, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for kudu/127.24.255.60@KRBTEST.COM
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402039, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.60@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:53:59.397917 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 6602
I20260926 05:53:59.398020 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401881171543-25596-0/minicluster-data/master-2/wal/instance
I20260926 05:53:59.398255 25596 external_mini_cluster.cc:1428] Setting key fcb98a1a9faeb23532fec8fa5e409b30
I20260926 05:53:59.400494 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.391527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35547 (local address 127.0.0.1:49202)
0926 05:53:59.391590 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.391602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:59.391660 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:59.392506 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:59.392507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:59.392682 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:59.392738 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.392740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.394000 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.394001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.394758 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.394761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.395310 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.395311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.395360 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.395795 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:59.395799 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:59.397175 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:59.399209 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.399210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.399210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.399462 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.399698 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.399699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.399699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.399762 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.400198 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:59.400199 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:59.400235 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 05:53:59.400330 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 05:53:59.400422 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:53:59.400671 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.393166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.24.255.62:38455)
0926 05:53:59.393211 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.393213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.393375 (+ 162us) server_negotiation.cc:407] Connection header received
0926 05:53:59.393405 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.393406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.393416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.393478 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.393902 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.394447 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.395308 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.395537 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.395593 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.397727 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.397728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.397729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.397736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.399461 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.399876 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.399877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.399878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.399923 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.400083 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.400084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.400085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.400279 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.400343 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.400534 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.400601 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:53:59.400905 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.393121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38455 (local address 127.0.0.1:34448)
0926 05:53:59.393268 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.393309 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:59.393373 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:59.393493 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:59.393494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:59.393734 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:59.393831 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.393835 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.394537 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.394539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.395232 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.395234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.395620 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.395621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.395704 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.396084 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:59.396093 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:59.397635 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:59.399561 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.399568 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.399571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.399819 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.399977 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.399977 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.399977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.400034 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.400342 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:59.400346 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:59.400412 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 05:53:59.400509 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 05:53:59.400672 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260926 05:53:59.401428 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.391685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.24.255.60:35547)
0926 05:53:59.391858 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.391866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.391890 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:53:59.392318 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.392324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.392370 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.392482 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.392855 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.393923 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.394837 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.395235 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.395333 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.397254 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.397262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.397263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.397269 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.399143 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.399535 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.399538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.399538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.399638 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.399824 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.399824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.399825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.400130 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.400208 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.401282 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.401362 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260926 05:53:59.401700 6670 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:53:59.403299 6517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007: Bootstrap starting.
I20260926 05:53:59.403923 6517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007: Neither blocks nor log segments found. Creating new log.
I20260926 05:53:59.404203 6517 log.cc:824] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007: Log is configured to *not* fsync() on all Append() calls
I20260926 05:53:59.404975 6517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007: No bootstrap required, opened a new log
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402039, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.60@KRBTEST.COM for kudu/127.24.255.61@KRBTEST.COM
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.60@KRBTEST.COM
I20260926 05:53:59.407135 6517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } }
I20260926 05:53:59.407294 6517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:53:59.407336 6517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202ead50c74c4e23bf0955dc65c54007, State: Initialized, Role: FOLLOWER
I20260926 05:53:59.407464 6517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } }
I20260926 05:53:59.407816 6517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:53:59.407770 6677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } } }
I20260926 05:53:59.408115 6677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:53:59.409308 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.401978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38439 (local address 127.0.0.1:57564)
0926 05:53:59.402166 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.402176 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:59.402221 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:59.402330 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:59.402331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:59.402451 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:59.402496 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.402497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.403078 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.403079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.403898 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.403900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.404258 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.404259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.404323 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.404721 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:59.404726 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:59.406235 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:59.407925 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.407926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.407926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.408187 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.408390 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.408391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.408391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.408452 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.408943 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:59.408944 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:59.409005 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 05:53:59.409092 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 05:53:59.409209 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260926 05:53:59.409534 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.402125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.24.255.61:38439)
0926 05:53:59.402179 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.402181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.402208 (+ 27us) server_negotiation.cc:407] Connection header received
0926 05:53:59.402276 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.402277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.402287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.402330 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.402567 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.403008 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.404010 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.404198 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.404254 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.406326 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.406328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.406329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.406338 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.407844 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.408260 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.408262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.408262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.408322 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.408513 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.408513 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.408514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.408799 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.408895 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.409344 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.409432 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:53:59.410086 6670 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:53:59.411309 6675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.398635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.24.255.60:35547)
0926 05:53:59.398813 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.398815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.398824 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:53:59.398880 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.398881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.398892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.398965 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.399632 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.400307 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.401295 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.401648 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.401710 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.407944 (+ 6234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.407946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.407946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.407953 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.409941 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.410370 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.410371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.410371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.410502 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.410690 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.410691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.410691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.410812 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.410968 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.411097 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.411178 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
W20260926 05:53:59.412293 6688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:53:59.412341 6688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:53:59.411128 6449 master_runner.cc:186] Error getting master registration for 127.24.255.61:38439: OK, instance_id { permanent_uuid: "e711288ccf75487885965714ae71f6ab" instance_seqno: 1790402039224756 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:53:59.413424 6449 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402039, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.60@KRBTEST.COM for kudu/127.24.255.60@KRBTEST.COM
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:53:59.417227 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.410400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.24.255.60:35547)
0926 05:53:59.410540 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.410542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.410548 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:53:59.410565 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.410566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.410574 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.410622 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.410883 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.411573 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.412506 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.412775 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.412827 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.414887 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.414889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.414889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.414896 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.416220 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.416505 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.416506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.416506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.416559 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.416711 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.416712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.416712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.416855 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.416912 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.417037 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.417135 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":122}
I20260926 05:53:59.417532 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.410353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35547 (local address 127.0.0.1:49218)
0926 05:53:59.410437 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.410446 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:59.410480 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:59.410624 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:59.410624 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:59.410770 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:59.410810 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.410812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.411697 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.411698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.412423 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.412426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.412872 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.412872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.412919 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.413341 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:59.413345 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:59.414812 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:59.416292 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.416293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.416293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.416447 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.416611 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.416611 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.416612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.416663 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.416912 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:59.416913 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:59.416952 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 05:53:59.417011 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 05:53:59.417436 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":325}
I20260926 05:53:59.419148 6670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8: Bootstrap starting.
I20260926 05:53:59.419905 6670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8: Neither blocks nor log segments found. Creating new log.
I20260926 05:53:59.420027 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.411874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.24.255.62:38455)
0926 05:53:59.411911 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.411912 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.412030 (+ 118us) server_negotiation.cc:407] Connection header received
0926 05:53:59.412049 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.412050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.412059 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.412098 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.412683 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.413323 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.414186 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.414451 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.414515 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.417163 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.417165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.417165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.417172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.418756 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.419231 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.419232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.419232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.419291 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.419498 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.419499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.419499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.419638 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.419812 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.419864 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.419944 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:53:59.420262 6670 log.cc:824] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:53:59.421149 6670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8: No bootstrap required, opened a new log
I20260926 05:53:59.422950 6670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } }
I20260926 05:53:59.423099 6670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:53:59.423138 6670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c6146929744a33af9090241b118be8, State: Initialized, Role: FOLLOWER
I20260926 05:53:59.423264 6670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } }
I20260926 05:53:59.423564 6670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:53:59.423985 6691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } } }
I20260926 05:53:59.424166 6691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:53:59.425525 6702 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:53:59.425631 6702 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.61@KRBTEST.COM
I20260926 05:53:59.430516 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.421049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.24.255.61:38439)
0926 05:53:59.421089 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.421091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.421098 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:53:59.421123 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.421123 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.421133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.421190 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.421799 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.422528 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.423460 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.423740 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.423812 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.426712 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.426714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.426714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.426723 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.428411 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.428830 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.428831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.428831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.428888 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.429076 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.429077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.429077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.429215 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.429839 (+ 624us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.430325 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.430407 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:53:59.428648 6602 master_runner.cc:186] Error getting master registration for 127.24.255.61:38439: OK, instance_id { permanent_uuid: "e711288ccf75487885965714ae71f6ab" instance_seqno: 1790402039224756 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:53:59.431000 6602 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:53:59.432298 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:53:59.441990 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.433863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.24.255.61:38439)
0926 05:53:59.433902 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.433904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.434057 (+ 153us) server_negotiation.cc:407] Connection header received
0926 05:53:59.434187 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.434188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.434199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.434227 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.435022 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.435768 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.436902 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.437346 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.437395 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.439151 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.439152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.439153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.439161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.440756 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.441156 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.441157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.441158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.441210 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.441395 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.441396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.441396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.441522 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.441593 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.441784 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.441870 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:53:59.447047 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.434205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.24.255.60:35547)
0926 05:53:59.434249 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.434251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.438259 (+ 4008us) server_negotiation.cc:407] Connection header received
0926 05:53:59.438364 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.438365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.438377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.438407 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.439077 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.440395 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.441426 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.441732 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.441775 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.443886 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.443897 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.443897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.443906 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.445519 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.446185 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.446186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.446186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.446250 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.446445 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.446446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.446446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.446571 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.446661 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.446849 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.446944 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:53:59.449695 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.434409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34466 (local address 127.24.255.62:38455)
0926 05:53:59.434449 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.434450 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.442683 (+ 8233us) server_negotiation.cc:407] Connection header received
0926 05:53:59.442785 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.442786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.442799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.442827 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.443543 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.444292 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.445260 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.445560 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.445599 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.446892 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.446893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.446894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.446901 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.448506 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.448867 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.448868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.448868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.448918 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.449115 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.449116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.449116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.449238 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.449305 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.449511 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.449595 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:53:59.470000 6594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } attempt: 3
Sep 26 05:53:59 dist-test-slave-7fr8 krb5kdc[6436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402039, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.61@KRBTEST.COM for kudu/127.24.255.60@KRBTEST.COM
I20260926 05:53:59.478823 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.470389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:35547 (local address 127.0.0.1:49236)
0926 05:53:59.470448 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 05:53:59.470459 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:53:59.470497 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:53:59.470650 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:53:59.470650 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:53:59.470841 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:53:59.470894 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.470896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.471875 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.471877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.472732 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:53:59.472735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.473227 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.473228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:53:59.473289 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.474180 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:53:59.474188 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:53:59.475779 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:53:59.477657 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.477659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.477659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.477884 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.478135 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:53:59.478136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:53:59.478136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:53:59.478206 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:53:59.478488 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:53:59.478489 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:53:59.478544 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 05:53:59.478608 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:53:59.478719 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 05:53:59.478823 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:53:59.470470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.24.255.60:35547)
0926 05:53:59.470521 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:53:59.470523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:53:59.470532 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:53:59.470559 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:53:59.470560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:53:59.470570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:53:59.470638 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:53:59.470987 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.471799 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.472819 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:53:59.473135 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:53:59.473221 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:53:59.475892 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:53:59.475893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:53:59.475894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:53:59.475903 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:53:59.477558 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.477970 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.477971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.477972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.478036 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:53:59.478266 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:53:59.478266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:53:59.478267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:53:59.478415 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:53:59.478477 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:53:59.478640 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 05:53:59.478723 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:53:59.480919 6594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab: Bootstrap starting.
I20260926 05:53:59.481946 6594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab: Neither blocks nor log segments found. Creating new log.
I20260926 05:53:59.482297 6594 log.cc:824] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab: Log is configured to *not* fsync() on all Append() calls
I20260926 05:53:59.483263 6594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab: No bootstrap required, opened a new log
I20260926 05:53:59.485241 6594 raft_consensus.cc:348] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } }
I20260926 05:53:59.485396 6594 raft_consensus.cc:374] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:53:59.485436 6594 raft_consensus.cc:729] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e711288ccf75487885965714ae71f6ab, State: Initialized, Role: FOLLOWER
I20260926 05:53:59.485559 6594 consensus_queue.cc:260] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } }
I20260926 05:53:59.485913 6594 sys_catalog.cc:564] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:53:59.485878 6712 sys_catalog.cc:455] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } } }
I20260926 05:53:59.486071 6712 sys_catalog.cc:458] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:53:59.487807 6723 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:53:59.487916 6723 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:53:59.489485 6691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:53:59.489586 6691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } }
I20260926 05:53:59.489825 6691 leader_election.cc:288] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 202ead50c74c4e23bf0955dc65c54007 (127.24.255.62:38455), e711288ccf75487885965714ae71f6ab (127.24.255.61:38439)
I20260926 05:53:59.490087 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91c6146929744a33af9090241b118be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e711288ccf75487885965714ae71f6ab" is_pre_election: true
I20260926 05:53:59.490082 6491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91c6146929744a33af9090241b118be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202ead50c74c4e23bf0955dc65c54007" is_pre_election: true
I20260926 05:53:59.490242 6568 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c6146929744a33af9090241b118be8 in term 0.
I20260926 05:53:59.490242 6491 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c6146929744a33af9090241b118be8 in term 0.
I20260926 05:53:59.490432 6620 leader_election.cc:302] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202ead50c74c4e23bf0955dc65c54007, 91c6146929744a33af9090241b118be8; no voters:
I20260926 05:53:59.490578 6691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:53:59.490628 6691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:53:59.490653 6691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:53:59.491503 6691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } }
I20260926 05:53:59.491636 6691 leader_election.cc:288] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [CANDIDATE]: Term 1 election: Requested vote from peers 202ead50c74c4e23bf0955dc65c54007 (127.24.255.62:38455), e711288ccf75487885965714ae71f6ab (127.24.255.61:38439)
I20260926 05:53:59.491819 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91c6146929744a33af9090241b118be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e711288ccf75487885965714ae71f6ab"
I20260926 05:53:59.491896 6568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:53:59.492758 6568 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c6146929744a33af9090241b118be8 in term 1.
I20260926 05:53:59.492944 6618 leader_election.cc:302] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c6146929744a33af9090241b118be8, e711288ccf75487885965714ae71f6ab; no voters:
I20260926 05:53:59.493168 6491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91c6146929744a33af9090241b118be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202ead50c74c4e23bf0955dc65c54007"
I20260926 05:53:59.493249 6491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:53:59.494836 6525 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260926 05:53:59.494925 6525 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:53:59.494081 6491 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c6146929744a33af9090241b118be8 in term 1.
I20260926 05:53:59.495642 6691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:53:59.495813 6691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [term 1 LEADER]: Becoming Leader. State: Replica: 91c6146929744a33af9090241b118be8, State: Running, Role: LEADER
I20260926 05:53:59.495914 6691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } }
I20260926 05:53:59.496346 6691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91c6146929744a33af9090241b118be8. Latest consensus state: current_term: 1 leader_uuid: "91c6146929744a33af9090241b118be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } } }
I20260926 05:53:59.496429 6691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:53:59.496636 6727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:53:59.496811 6727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:53:59.499065 6568 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [term 1 FOLLOWER]: Refusing update from remote peer 91c6146929744a33af9090241b118be8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:53:59.499390 6691 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:53:59.500707 6712 sys_catalog.cc:455] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91c6146929744a33af9090241b118be8. Latest consensus state: current_term: 1 leader_uuid: "91c6146929744a33af9090241b118be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } } }
I20260926 05:53:59.500821 6712 sys_catalog.cc:458] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:53:59.501233 6491 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [term 1 FOLLOWER]: Refusing update from remote peer 91c6146929744a33af9090241b118be8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:53:59.501537 6691 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:53:59.503723 6677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91c6146929744a33af9090241b118be8. Latest consensus state: current_term: 1 leader_uuid: "91c6146929744a33af9090241b118be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } } }
I20260926 05:53:59.503845 6677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:53:59.506870 6691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91c6146929744a33af9090241b118be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } } }
I20260926 05:53:59.507009 6691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:53:59.507071 6729 mvcc.cc:205] Tried to move back new op lower bound from 7333486753785589760 to 7333486753776017408. Current Snapshot: MvccSnapshot[applied={T|T < 7333486753785589760}]
I20260926 05:53:59.507145 6712 sys_catalog.cc:455] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91c6146929744a33af9090241b118be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } } }
I20260926 05:53:59.507225 6712 sys_catalog.cc:458] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:53:59.507318 6728 mvcc.cc:205] Tried to move back new op lower bound from 7333486753785589760 to 7333486753776017408. Current Snapshot: MvccSnapshot[applied={T|T < 7333486753785589760}]
I20260926 05:53:59.515225 6727 catalog_manager.cc:1347] Generated new cluster ID: e7f706134a5a4315a59d88b435e63a5b
I20260926 05:53:59.515314 6727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:53:59.530652 6734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91c6146929744a33af9090241b118be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } } }
I20260926 05:53:59.531250 6734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:53:59.531858 6677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91c6146929744a33af9090241b118be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202ead50c74c4e23bf0955dc65c54007" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38455 } } peers { permanent_uuid: "e711288ccf75487885965714ae71f6ab" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38439 } } peers { permanent_uuid: "91c6146929744a33af9090241b118be8" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 35547 } } }
I20260926 05:53:59.531963 6677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:53:59.533185 6731 mvcc.cc:205] Tried to move back new op lower bound from 7333486753785589760 to 7333486753776017408. Current Snapshot: MvccSnapshot[applied={T|T < 7333486753785589760}]
I20260926 05:53:59.553442 6727 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:53:59.554296 6727 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:53:59.560192 6727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91c6146929744a33af9090241b118be8: Generated new TSK 0
I20260926 05:53:59.560438 6727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:54:00.412880 6688 catalog_manager.cc:1259] Loaded cluster ID: e7f706134a5a4315a59d88b435e63a5b
I20260926 05:54:00.412961 6688 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007: loading cluster ID for follower catalog manager: success
I20260926 05:54:00.413795 6688 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007: acquiring CA information for follower catalog manager: success
I20260926 05:54:00.414149 6688 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 202ead50c74c4e23bf0955dc65c54007: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:54:00.488440 6723 catalog_manager.cc:1259] Loaded cluster ID: e7f706134a5a4315a59d88b435e63a5b
I20260926 05:54:00.488520 6723 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab: loading cluster ID for follower catalog manager: success
I20260926 05:54:00.489295 6723 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab: acquiring CA information for follower catalog manager: success
I20260926 05:54:00.489590 6723 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e711288ccf75487885965714ae71f6ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:54:00.624919 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 6449
I20260926 05:54:00.630000 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 6525
I20260926 05:54:00.634500 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 6602
2026-09-26T05:54:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2830 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:54:00 dist-test-slave-7fr8 krb5kdc[6743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:54:00 dist-test-slave-7fr8 krb5kdc[6743](info): set up 2 sockets
Sep 26 05:54:00 dist-test-slave-7fr8 krb5kdc[6743](info): commencing operation
krb5kdc: starting...
W20260926 05:54:01.610759 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:54:01 dist-test-slave-7fr8 krb5kdc[6743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:54:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:54:01.739488 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:45583
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:45583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:54:01.827065 6756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:01.827329 6756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:01.827355 6756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:01.828722 6756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:54:01.828788 6756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:01.828802 6756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:01.828814 6756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:54:01.828825 6756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:54:01.830770 6756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:45583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:45583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:01.831058 6756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:01.831298 6756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:01.837824 6762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:01.837843 6761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:01.837827 6764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:01.837918 6756 server_base.cc:1034] running on GCE node
I20260926 05:54:01.838330 6756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:01.838665 6756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:01.839841 6756 hybrid_clock.cc:648] HybridClock initialized: now 1790402041839821 us; error 37 us; skew 500 ppm
Sep 26 05:54:01 dist-test-slave-7fr8 krb5kdc[6743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402041, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:01.842859 6756 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:54:01.843344 6756 webserver.cc:460] Webserver started at http://127.24.255.62:34493/ using document root <none> and password file <none>
I20260926 05:54:01.843559 6756 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:01.843609 6756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:01.843737 6756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:54:01.844983 6756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "b6c6d2e0515d4ea7b2337ffcb2499fe8"
format_stamp: "Formatted at 2026-09-26 05:54:01 on dist-test-slave-7fr8"
server_key: "6efaad86d89b15353eec7cf9346e18aa"
server_key_iv: "c9942284db25984458c0ca3a285067d6"
server_key_version: "encryptionkey@0"
I20260926 05:54:01.845345 6756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "b6c6d2e0515d4ea7b2337ffcb2499fe8"
format_stamp: "Formatted at 2026-09-26 05:54:01 on dist-test-slave-7fr8"
server_key: "6efaad86d89b15353eec7cf9346e18aa"
server_key_iv: "c9942284db25984458c0ca3a285067d6"
server_key_version: "encryptionkey@0"
I20260926 05:54:01.846787 6756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:54:01.847693 6771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:01.847865 6756 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:01.847963 6756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "b6c6d2e0515d4ea7b2337ffcb2499fe8"
format_stamp: "Formatted at 2026-09-26 05:54:01 on dist-test-slave-7fr8"
server_key: "6efaad86d89b15353eec7cf9346e18aa"
server_key_iv: "c9942284db25984458c0ca3a285067d6"
server_key_version: "encryptionkey@0"
I20260926 05:54:01.848037 6756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:01.880125 6756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:01.882905 6756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:01.883054 6756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:01.887880 6756 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:45583
I20260926 05:54:01.887966 6823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:45583 every 8 connection(s)
I20260926 05:54:01.888345 6756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:54:01.889220 6824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:01.891789 6824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8: Bootstrap starting.
I20260926 05:54:01.892467 6824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:01.892880 6824 log.cc:824] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:01.893744 6824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8: No bootstrap required, opened a new log
I20260926 05:54:01.895520 6824 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c6d2e0515d4ea7b2337ffcb2499fe8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45583 } }
I20260926 05:54:01.895684 6824 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:01.895707 6824 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c6d2e0515d4ea7b2337ffcb2499fe8, State: Initialized, Role: FOLLOWER
I20260926 05:54:01.895794 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 6756
I20260926 05:54:01.895845 6824 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c6d2e0515d4ea7b2337ffcb2499fe8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45583 } }
I20260926 05:54:01.895893 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:54:01.895931 6824 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:54:01.895970 6824 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:54:01.896019 6824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:01.896117 25596 external_mini_cluster.cc:1428] Setting key 44d087acf2b13f1f14c656d31e443280
I20260926 05:54:01.896780 6824 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c6d2e0515d4ea7b2337ffcb2499fe8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45583 } }
I20260926 05:54:01.896883 6824 leader_election.cc:302] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6c6d2e0515d4ea7b2337ffcb2499fe8; no voters:
I20260926 05:54:01.897068 6824 leader_election.cc:288] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:54:01.897332 6824 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:54:01.897324 6829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:54:01.897622 6829 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [term 1 LEADER]: Becoming Leader. State: Replica: b6c6d2e0515d4ea7b2337ffcb2499fe8, State: Running, Role: LEADER
I20260926 05:54:01.897724 6829 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c6d2e0515d4ea7b2337ffcb2499fe8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45583 } }
I20260926 05:54:01.898131 6831 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6c6d2e0515d4ea7b2337ffcb2499fe8. Latest consensus state: current_term: 1 leader_uuid: "b6c6d2e0515d4ea7b2337ffcb2499fe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c6d2e0515d4ea7b2337ffcb2499fe8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45583 } } }
I20260926 05:54:01.898213 6831 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:54:01.898558 6830 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6c6d2e0515d4ea7b2337ffcb2499fe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c6d2e0515d4ea7b2337ffcb2499fe8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 45583 } } }
I20260926 05:54:01.898627 6830 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8 [sys.catalog]: This master's current role is: LEADER
W20260926 05:54:01.899317 6844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:54:01.899372 6844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:54:01.899428 6838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:54:01.899595 6838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:54:01.901858 6838 catalog_manager.cc:1347] Generated new cluster ID: f19ec449186548e1b1786400dcc17779
I20260926 05:54:01.901914 6838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:54:01 dist-test-slave-7fr8 krb5kdc[6743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:54:01.907097 6838 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:54:01.907429 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:01.896925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.24.255.62:45583)
0926 05:54:01.897114 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:01.897126 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:01.897169 (+ 43us) server_negotiation.cc:407] Connection header received
0926 05:54:01.897296 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:01.897304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:01.897364 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:01.898342 (+ 978us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:01.898497 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:01.899764 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:01.900652 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:01.901273 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:01.901398 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:01.903739 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:01.903743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:01.903744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:01.903756 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:01.905714 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:01.906065 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:01.906069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:01.906069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:01.906212 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:01.906392 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:01.906393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:01.906393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:01.906649 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:01.906813 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:01.906975 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 05:54:01.907139 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260926 05:54:01.907795 6838 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:54:01.908375 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:54:01.915081 6838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6c6d2e0515d4ea7b2337ffcb2499fe8: Generated new TSK 0
I20260926 05:54:01.915438 6838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:54:01.963289 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 6756
2026-09-26T05:54:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1327 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:54:02 dist-test-slave-7fr8 krb5kdc[6851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:54:02 dist-test-slave-7fr8 krb5kdc[6851](info): set up 2 sockets
Sep 26 05:54:02 dist-test-slave-7fr8 krb5kdc[6851](info): commencing operation
krb5kdc: starting...
W20260926 05:54:02.950556 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:54:03 dist-test-slave-7fr8 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:54:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:54:03.078339 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:35393
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:35393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:54:03.166205 6864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:03.166765 6864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:03.166849 6864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:03.168252 6864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:54:03.168315 6864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:03.168335 6864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:03.168354 6864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:54:03.168370 6864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:54:03.170092 6864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:35393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:35393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:03.170409 6864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:03.170648 6864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:03.175819 6870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:03.175824 6872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:03.175823 6869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:03.176029 6864 server_base.cc:1034] running on GCE node
I20260926 05:54:03.176295 6864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:03.176573 6864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:03.177748 6864 hybrid_clock.cc:648] HybridClock initialized: now 1790402043177728 us; error 35 us; skew 500 ppm
Sep 26 05:54:03 dist-test-slave-7fr8 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402043, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:03.180583 6864 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:54:03.181077 6864 webserver.cc:460] Webserver started at http://127.24.255.62:44755/ using document root <none> and password file <none>
I20260926 05:54:03.181321 6864 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:03.181380 6864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:03.181504 6864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:54:03.182655 6864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "8b8f701658e040b8a114896b4175b52a"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "348ac58b300a568c8d8e901d424b2711"
server_key_iv: "755095b338d098f0ee21e4ee584f1f27"
server_key_version: "encryptionkey@0"
I20260926 05:54:03.182987 6864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "8b8f701658e040b8a114896b4175b52a"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "348ac58b300a568c8d8e901d424b2711"
server_key_iv: "755095b338d098f0ee21e4ee584f1f27"
server_key_version: "encryptionkey@0"
I20260926 05:54:03.184367 6864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:54:03.185737 6879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:03.185945 6864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:03.186039 6864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "8b8f701658e040b8a114896b4175b52a"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "348ac58b300a568c8d8e901d424b2711"
server_key_iv: "755095b338d098f0ee21e4ee584f1f27"
server_key_version: "encryptionkey@0"
I20260926 05:54:03.186144 6864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:03.204234 6864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:03.206496 6864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:03.206614 6864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:03.210714 6864 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:35393
I20260926 05:54:03.210857 6931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:35393 every 8 connection(s)
I20260926 05:54:03.211131 6864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:54:03.212427 6932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:03.215378 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 6864
I20260926 05:54:03.215477 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:54:03.215723 25596 external_mini_cluster.cc:1428] Setting key 1ea0efa11a207ca6a7a4ba3768610d3b
I20260926 05:54:03.217068 6932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Bootstrap starting.
I20260926 05:54:03.217667 6932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:03.217927 6932 log.cc:824] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:03.218606 6932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: No bootstrap required, opened a new log
I20260926 05:54:03.220403 6932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } }
I20260926 05:54:03.220567 6932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:03.220602 6932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8f701658e040b8a114896b4175b52a, State: Initialized, Role: FOLLOWER
I20260926 05:54:03.220739 6932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } }
I20260926 05:54:03.220845 6932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:54:03.220881 6932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:54:03.220935 6932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:03.221655 6932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } }
I20260926 05:54:03.221771 6932 leader_election.cc:302] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b8f701658e040b8a114896b4175b52a; no voters:
I20260926 05:54:03.221971 6932 leader_election.cc:288] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:54:03.222165 6937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:54:03.222363 6937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 1 LEADER]: Becoming Leader. State: Replica: 8b8f701658e040b8a114896b4175b52a, State: Running, Role: LEADER
I20260926 05:54:03.222492 6932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:54:03.222507 6937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } }
I20260926 05:54:03.222898 6937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b8f701658e040b8a114896b4175b52a. Latest consensus state: current_term: 1 leader_uuid: "8b8f701658e040b8a114896b4175b52a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } } }
I20260926 05:54:03.222971 6937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: This master's current role is: LEADER
I20260926 05:54:03.223109 6938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b8f701658e040b8a114896b4175b52a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } } }
I20260926 05:54:03.223187 6938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: This master's current role is: LEADER
Sep 26 05:54:03 dist-test-slave-7fr8 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:54:03.224128 6952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:54:03.224721 6952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:54:03.226818 6952 catalog_manager.cc:1347] Generated new cluster ID: 610c8d3c949e4198bdab12aa34776db1
I20260926 05:54:03.226903 6952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:54:03.228626 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.216630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.24.255.62:35393)
0926 05:54:03.217277 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:03.217289 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:03.217331 (+ 42us) server_negotiation.cc:407] Connection header received
0926 05:54:03.217826 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:03.217833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:03.217886 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:03.218300 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:03.218932 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.219791 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.220759 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.221252 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.221362 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.224496 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:03.224501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:03.224502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:03.224517 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:03.226536 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:03.226970 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:03.226973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:03.226974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:03.227079 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:03.227320 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:03.227321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:03.227322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:03.227815 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:03.228102 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:03.228210 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 05:54:03.228360 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":109,"threads_started":1}
I20260926 05:54:03.256570 6952 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:54:03.257153 6952 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:54:03.263201 6952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Generated new TSK 0
I20260926 05:54:03.263433 6952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:54:03.294911 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35393
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:54:03.382409 6958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:03.382651 6958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:03.382678 6958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:03.383999 6958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:03.384061 6958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:03.384093 6958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:54:03.385744 6958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:03.386004 6958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:03.386289 6958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:03.391798 6964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:03.391798 6963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:03.392098 6958 server_base.cc:1034] running on GCE node
W20260926 05:54:03.391970 6966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:03.392355 6958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:03.392629 6958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:03.393812 6958 hybrid_clock.cc:648] HybridClock initialized: now 1790402043393764 us; error 54 us; skew 500 ppm
Sep 26 05:54:03 dist-test-slave-7fr8 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402043, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:03.396694 6958 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:54:03.397251 6958 webserver.cc:460] Webserver started at http://127.24.255.1:35959/ using document root <none> and password file <none>
I20260926 05:54:03.397488 6958 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:03.397532 6958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:03.397652 6958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:54:03.398810 6958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "fe6f885c67b14c259ba22f0b5d70c0bb"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "ee91f9207c24bbdedb3d2097e3a29985"
server_key_iv: "826f13fe95715f8b636acfeaecb502c2"
server_key_version: "encryptionkey@0"
I20260926 05:54:03.399138 6958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "fe6f885c67b14c259ba22f0b5d70c0bb"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "ee91f9207c24bbdedb3d2097e3a29985"
server_key_iv: "826f13fe95715f8b636acfeaecb502c2"
server_key_version: "encryptionkey@0"
I20260926 05:54:03.400645 6958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:54:03.402072 6973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:03.402412 6958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:03.402577 6958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "fe6f885c67b14c259ba22f0b5d70c0bb"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "ee91f9207c24bbdedb3d2097e3a29985"
server_key_iv: "826f13fe95715f8b636acfeaecb502c2"
server_key_version: "encryptionkey@0"
I20260926 05:54:03.402691 6958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:03.428440 6958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:03.431576 6958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:03.431721 6958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:03.431977 6958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:03.432425 6958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:54:03.432468 6958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:03.432492 6958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:54:03.432503 6958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:03.439826 6958 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:38011
I20260926 05:54:03.439865 7086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:38011 every 8 connection(s)
I20260926 05:54:03.440236 6958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:54:03.441293 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 6958
I20260926 05:54:03.441454 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:54:03.441648 25596 external_mini_cluster.cc:1428] Setting key c4bbd30a560e91f4f1170abdc988b3af
Sep 26 05:54:03 dist-test-slave-7fr8 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402043, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
I20260926 05:54:03.451834 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.441026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:57257 (local address 127.24.255.62:35393)
0926 05:54:03.441111 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:03.441113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:03.441308 (+ 195us) server_negotiation.cc:407] Connection header received
0926 05:54:03.441946 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:03.441947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:03.441959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:03.442077 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:03.443220 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.443942 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.445072 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.445380 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.445462 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.448325 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:03.448327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:03.448327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:03.448337 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:03.449877 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:03.450391 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:03.450392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:03.450392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:03.450617 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:03.451074 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:03.451075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:03.451075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:03.451216 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:03.451283 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:03.451629 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 05:54:03.451735 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 05:54:03.452065 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.441021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:57257)
0926 05:54:03.441170 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:03.441215 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:03.441795 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:03.442200 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:03.442205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:03.442911 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:03.443118 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.443123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.444072 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.444073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.444968 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.444971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.445684 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.445687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.445929 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.446715 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:03.446727 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:03.448196 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:03.449985 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:03.449996 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:03.449998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:03.450310 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:03.450807 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:03.450808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:03.450809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:03.450974 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:03.451291 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:03.451298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:03.451482 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 05:54:03.451576 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 05:54:03.451760 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260926 05:54:03.453003 7087 heartbeater.cc:347] Connected to a master server at 127.24.255.62:35393
I20260926 05:54:03.453141 7087 heartbeater.cc:464] Registering TS with master...
I20260926 05:54:03.453330 7087 heartbeater.cc:511] Master 127.24.255.62:35393 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
I20260926 05:54:03.454010 6896 ts_manager.cc:194] Registered new tserver with Master: fe6f885c67b14c259ba22f0b5d70c0bb (127.24.255.1:38011)
I20260926 05:54:03.454954 6896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:57257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:54:03.473615 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35393
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:54:03.556003 7092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:03.556233 7092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:03.556259 7092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:03.557432 7092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:03.557478 7092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:03.557505 7092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:54:03.559002 7092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:03.559211 7092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:03.559393 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:03.564574 7097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:03.564622 7098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:03.564651 7092 server_base.cc:1034] running on GCE node
W20260926 05:54:03.564595 7100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:03.565001 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:03.565253 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:03.566424 7092 hybrid_clock.cc:648] HybridClock initialized: now 1790402043566413 us; error 29 us; skew 500 ppm
Sep 26 05:54:03 dist-test-slave-7fr8 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402043, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:03.569079 7092 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:54:03.569588 7092 webserver.cc:460] Webserver started at http://127.24.255.2:33267/ using document root <none> and password file <none>
I20260926 05:54:03.569792 7092 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:03.569837 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:03.569948 7092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:54:03.571154 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "b06214c9867b494c84313036fed37dcf"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "b7054c158ad0f29ee5fa5bc964ae3aae"
server_key_iv: "ddc912c8a8aab03b865f35e74a21cd2f"
server_key_version: "encryptionkey@0"
I20260926 05:54:03.571513 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "b06214c9867b494c84313036fed37dcf"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "b7054c158ad0f29ee5fa5bc964ae3aae"
server_key_iv: "ddc912c8a8aab03b865f35e74a21cd2f"
server_key_version: "encryptionkey@0"
I20260926 05:54:03.572897 7092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:54:03.573784 7107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:03.574031 7092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:03.574158 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "b06214c9867b494c84313036fed37dcf"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "b7054c158ad0f29ee5fa5bc964ae3aae"
server_key_iv: "ddc912c8a8aab03b865f35e74a21cd2f"
server_key_version: "encryptionkey@0"
I20260926 05:54:03.574234 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:03.593019 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:03.595799 7092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:03.595975 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:03.596232 7092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:03.596572 7092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:54:03.596606 7092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:03.596637 7092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:54:03.596662 7092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:03.602646 7092 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:44457
I20260926 05:54:03.602703 7220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:44457 every 8 connection(s)
I20260926 05:54:03.603008 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:54:03.608979 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 7092
I20260926 05:54:03.609081 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:54:03.609251 25596 external_mini_cluster.cc:1428] Setting key 9d2f663fa0fad8b4cfd071e34e841084
Sep 26 05:54:03 dist-test-slave-7fr8 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402043, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:54:03.614041 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.603785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:60599 (local address 127.24.255.62:35393)
0926 05:54:03.603849 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:03.603852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:03.604020 (+ 168us) server_negotiation.cc:407] Connection header received
0926 05:54:03.604567 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:03.604569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:03.604580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:03.604671 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:03.605753 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.606445 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.607594 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.607848 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.607954 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.610741 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:03.610743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:03.610743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:03.610752 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:03.612419 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:03.613039 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:03.613041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:03.613041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:03.613107 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:03.613360 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:03.613361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:03.613361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:03.613491 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:03.613549 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:03.613844 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 05:54:03.613936 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:54:03.614255 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.603719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:60599)
0926 05:54:03.603899 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:03.603940 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:03.604456 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:03.604741 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:03.604745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:03.605393 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:03.605628 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.605633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.606566 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.606567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.607425 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.607428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.607997 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.607998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.608152 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.608894 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:03.608903 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:03.610631 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:03.612652 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:03.612660 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:03.612663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:03.612915 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:03.613172 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:03.613173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:03.613173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:03.613299 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:03.613550 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:03.613558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:03.613718 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 05:54:03.613813 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 05:54:03.613990 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260926 05:54:03.614681 7221 heartbeater.cc:347] Connected to a master server at 127.24.255.62:35393
I20260926 05:54:03.614773 7221 heartbeater.cc:464] Registering TS with master...
I20260926 05:54:03.614965 7221 heartbeater.cc:511] Master 127.24.255.62:35393 requested a full tablet report, sending...
I20260926 05:54:03.615336 6896 ts_manager.cc:194] Registered new tserver with Master: b06214c9867b494c84313036fed37dcf (127.24.255.2:44457)
I20260926 05:54:03.615715 6896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:60599
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:54:03.637557 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35393
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:54:03.723493 7226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:03.723727 7226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:03.723762 7226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:03.725199 7226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:03.725256 7226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:03.725296 7226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:54:03.727015 7226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:03.727279 7226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:03.727502 7226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:03.732865 7231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:03.732883 7232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:03.732955 7226 server_base.cc:1034] running on GCE node
W20260926 05:54:03.732903 7234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:03.733287 7226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:03.733592 7226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:03.734755 7226 hybrid_clock.cc:648] HybridClock initialized: now 1790402043734736 us; error 34 us; skew 500 ppm
Sep 26 05:54:03 dist-test-slave-7fr8 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402043, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:03.737432 7226 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:54:03.737937 7226 webserver.cc:460] Webserver started at http://127.24.255.3:36499/ using document root <none> and password file <none>
I20260926 05:54:03.738183 7226 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:03.738233 7226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:03.738355 7226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:54:03.739423 7226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "6dc9e846bba4ad644cafd68543b48882"
server_key_iv: "88b8aeb119e517c27e70cbc668582dbf"
server_key_version: "encryptionkey@0"
I20260926 05:54:03.739756 7226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "6dc9e846bba4ad644cafd68543b48882"
server_key_iv: "88b8aeb119e517c27e70cbc668582dbf"
server_key_version: "encryptionkey@0"
I20260926 05:54:03.741313 7226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:54:03.742316 7241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:03.742570 7226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:03.742671 7226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "6dc9e846bba4ad644cafd68543b48882"
server_key_iv: "88b8aeb119e517c27e70cbc668582dbf"
server_key_version: "encryptionkey@0"
I20260926 05:54:03.742738 7226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:03.757383 7226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:03.759696 7226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:03.759821 7226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:03.760016 7226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:03.760346 7226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:54:03.760381 7226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:03.760411 7226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:54:03.760435 7226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:03.767012 7226 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:45331
I20260926 05:54:03.767084 7354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:45331 every 8 connection(s)
I20260926 05:54:03.767395 7226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:54:03.773294 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 7226
I20260926 05:54:03.773417 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:54:03.773602 25596 external_mini_cluster.cc:1428] Setting key 47e3c26c918e874e6685fcaf699ea2a8
Sep 26 05:54:03 dist-test-slave-7fr8 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402043, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:54:03.777446 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.768152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:44317)
0926 05:54:03.768315 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:03.768358 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:03.768888 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:03.769125 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:03.769129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:03.769748 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:03.769978 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.769984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.770796 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.770797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.771656 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.771659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.772076 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.772077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.772204 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.772704 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:03.772714 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:03.774289 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:03.775943 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:03.775951 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:03.775953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:03.776195 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:03.776432 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:03.776433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:03.776433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:03.776604 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:03.776828 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:03.776831 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:03.776960 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 05:54:03.777062 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 05:54:03.777209 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260926 05:54:03.777474 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.768181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:44317 (local address 127.24.255.62:35393)
0926 05:54:03.768232 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:03.768234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:03.768435 (+ 201us) server_negotiation.cc:407] Connection header received
0926 05:54:03.768987 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:03.768988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:03.768998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:03.769060 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:03.770089 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.770690 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.771755 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.771972 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.772038 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.774422 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:03.774424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:03.774424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:03.774433 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:03.775847 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:03.776280 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:03.776282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:03.776282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:03.776342 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:03.776665 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:03.776666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:03.776666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:03.776783 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:03.776833 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:03.777190 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 05:54:03.777355 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:54:03.778216 7355 heartbeater.cc:347] Connected to a master server at 127.24.255.62:35393
I20260926 05:54:03.778319 7355 heartbeater.cc:464] Registering TS with master...
I20260926 05:54:03.778478 7355 heartbeater.cc:511] Master 127.24.255.62:35393 requested a full tablet report, sending...
I20260926 05:54:03.778906 6896 ts_manager.cc:194] Registered new tserver with Master: feb45481d05e4dcfa6cc77e4d5bf5f30 (127.24.255.3:45331)
I20260926 05:54:03.779299 6896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:44317
I20260926 05:54:03.785902 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:54:03.794860 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.787445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.24.255.62:35393)
0926 05:54:03.787543 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:03.787544 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:03.787555 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:54:03.787630 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:03.787631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:03.787641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:03.787688 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:03.788309 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.789082 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.789956 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.790354 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.790422 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.791935 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:03.791938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:03.791938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:03.791948 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:03.793376 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:03.793920 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:03.793923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:03.793924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:03.794039 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:03.794295 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:03.794297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:03.794297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:03.794494 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:03.794590 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:03.794697 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 05:54:03.794781 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 05:54:03.797577 6896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:54:03.799517 6896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:54:03.807940 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.803322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:45331 (local address 127.0.0.1:57722)
0926 05:54:03.803523 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:03.803570 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:03.803651 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:03.804476 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:03.804477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:03.804482 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:03.804565 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.804566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.805705 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.805706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.806783 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.806787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.807627 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.807628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.807696 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.807708 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 05:54:03.807781 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 05:54:03.807842 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":48,"thread_start_us":77,"threads_started":1}
I20260926 05:54:03.807948 7369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.803535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.24.255.3:45331)
0926 05:54:03.804167 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:03.804172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:03.804182 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:54:03.804268 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:03.804271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:03.804288 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:03.804376 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:03.804720 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.805560 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.806863 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.807530 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.807610 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.807675 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:03.807823 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 05:54:03.807872 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":112,"threads_started":1}
I20260926 05:54:03.808240 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.803324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:38011 (local address 127.0.0.1:59692)
0926 05:54:03.803740 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:03.803750 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:03.803806 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:03.804189 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:03.804193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:03.804195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:03.804282 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.804284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.805637 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.805639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.807093 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.807094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.807994 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.807995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.808063 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.808068 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:54:03.808118 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 05:54:03.808165 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260926 05:54:03.808727 7368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.803478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59692 (local address 127.24.255.1:38011)
0926 05:54:03.803984 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:03.803990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:03.804001 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:54:03.804047 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:03.804049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:03.804067 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:03.804172 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:03.804375 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.805527 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.807180 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.807898 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.808479 (+ 581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.808554 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:03.808593 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 05:54:03.808641 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":121,"threads_started":1}
I20260926 05:54:03.808802 7289 tablet_service.cc:1467] Processing CreateTablet for tablet d484755f550f410c8f0f19176f49676d (DEFAULT_TABLE table=test-table [id=bbb687dd7c674708af6c070d9a667956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:54:03.809223 7289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d484755f550f410c8f0f19176f49676d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:03.809290 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.803552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:44457 (local address 127.0.0.1:43578)
0926 05:54:03.803914 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:03.803922 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:03.803958 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:03.804274 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:03.804275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:03.804277 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:03.804446 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.804457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.805776 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.805776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.807090 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.807093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.809043 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.809044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.809108 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.809114 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:54:03.809174 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:54:03.809222 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":82,"thread_start_us":125,"threads_started":1}
I20260926 05:54:03.809326 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.803851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.24.255.2:44457)
0926 05:54:03.804122 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:03.804126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:03.804136 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:54:03.804174 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:03.804176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:03.804191 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:03.804265 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:03.804594 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.805694 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.807732 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.808913 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.808990 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.809049 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:03.809202 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 05:54:03.809250 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":156,"threads_started":1}
I20260926 05:54:03.809485 7021 tablet_service.cc:1467] Processing CreateTablet for tablet d484755f550f410c8f0f19176f49676d (DEFAULT_TABLE table=test-table [id=bbb687dd7c674708af6c070d9a667956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:54:03.809742 7021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d484755f550f410c8f0f19176f49676d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:03.812088 7372 tablet_bootstrap.cc:492] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30: Bootstrap starting.
I20260926 05:54:03.812834 7372 tablet_bootstrap.cc:654] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:03.812774 7155 tablet_service.cc:1467] Processing CreateTablet for tablet d484755f550f410c8f0f19176f49676d (DEFAULT_TABLE table=test-table [id=bbb687dd7c674708af6c070d9a667956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:54:03.812879 7373 tablet_bootstrap.cc:492] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Bootstrap starting.
I20260926 05:54:03.813143 7155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d484755f550f410c8f0f19176f49676d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:03.813400 7372 log.cc:824] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:03.813732 7373 tablet_bootstrap.cc:654] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:03.814394 7373 log.cc:824] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:03.814539 7372 tablet_bootstrap.cc:492] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30: No bootstrap required, opened a new log
I20260926 05:54:03.814605 7372 ts_tablet_manager.cc:1397] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:54:03.815368 7373 tablet_bootstrap.cc:492] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: No bootstrap required, opened a new log
I20260926 05:54:03.815443 7373 ts_tablet_manager.cc:1397] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:54:03.816398 7376 tablet_bootstrap.cc:492] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf: Bootstrap starting.
I20260926 05:54:03.816886 7372 raft_consensus.cc:348] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } } peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } }
I20260926 05:54:03.817034 7372 raft_consensus.cc:374] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:03.817063 7372 raft_consensus.cc:729] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb45481d05e4dcfa6cc77e4d5bf5f30, State: Initialized, Role: FOLLOWER
I20260926 05:54:03.817206 7376 tablet_bootstrap.cc:654] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:03.817133 7373 raft_consensus.cc:348] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } } peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } }
I20260926 05:54:03.817224 7372 consensus_queue.cc:260] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } } peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } }
I20260926 05:54:03.817343 7373 raft_consensus.cc:374] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:03.817396 7373 raft_consensus.cc:729] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6f885c67b14c259ba22f0b5d70c0bb, State: Initialized, Role: FOLLOWER
I20260926 05:54:03.817533 7355 heartbeater.cc:503] Master 127.24.255.62:35393 was elected leader, sending a full tablet report...
I20260926 05:54:03.817529 7372 ts_tablet_manager.cc:1428] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:54:03.817624 7373 consensus_queue.cc:260] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } } peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } }
I20260926 05:54:03.817739 7376 log.cc:824] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:03.817890 7373 ts_tablet_manager.cc:1428] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:54:03.818063 7087 heartbeater.cc:503] Master 127.24.255.62:35393 was elected leader, sending a full tablet report...
I20260926 05:54:03.818682 7376 tablet_bootstrap.cc:492] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf: No bootstrap required, opened a new log
I20260926 05:54:03.818750 7376 ts_tablet_manager.cc:1397] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:54:03.819934 7378 raft_consensus.cc:482] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:54:03.820093 7378 raft_consensus.cc:504] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } } peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } }
I20260926 05:54:03.820421 7376 raft_consensus.cc:348] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } } peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } }
I20260926 05:54:03.820533 7378 leader_election.cc:288] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feb45481d05e4dcfa6cc77e4d5bf5f30 (127.24.255.3:45331), b06214c9867b494c84313036fed37dcf (127.24.255.2:44457)
I20260926 05:54:03.820639 7376 raft_consensus.cc:374] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:03.820699 7376 raft_consensus.cc:729] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06214c9867b494c84313036fed37dcf, State: Initialized, Role: FOLLOWER
I20260926 05:54:03.820844 7376 consensus_queue.cc:260] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } } peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } }
I20260926 05:54:03.821164 7376 ts_tablet_manager.cc:1428] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:54:03.821372 7221 heartbeater.cc:503] Master 127.24.255.62:35393 was elected leader, sending a full tablet report...
I20260926 05:54:03.823761 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.820732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:44457 (local address 127.24.255.1:47727)
0926 05:54:03.820808 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:03.820820 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:03.820863 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:03.821038 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:03.821039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:03.821041 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:03.821115 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.821117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.821879 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.821881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.822778 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.822781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.823515 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.823516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.823593 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.823597 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:54:03.823642 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 05:54:03.823685 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 05:54:03.823758 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.820823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:47727 (local address 127.24.255.2:44457)
0926 05:54:03.820884 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:03.820886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:03.820919 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:54:03.820952 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:03.820953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:03.820965 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:03.821029 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:03.821220 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.821801 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.822866 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.823436 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.823505 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.823516 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:03.823655 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 05:54:03.823695 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 05:54:03.824059 7175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d484755f550f410c8f0f19176f49676d" candidate_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06214c9867b494c84313036fed37dcf" is_pre_election: true
I20260926 05:54:03.824200 7175 raft_consensus.cc:2454] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe6f885c67b14c259ba22f0b5d70c0bb in term 0.
I20260926 05:54:03.824433 6977 leader_election.cc:302] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b06214c9867b494c84313036fed37dcf, fe6f885c67b14c259ba22f0b5d70c0bb; no voters:
I20260926 05:54:03.824636 7378 raft_consensus.cc:2781] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:54:03.824788 7378 raft_consensus.cc:482] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:54:03.824823 7378 raft_consensus.cc:3037] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:03.825029 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.820766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:45331 (local address 127.24.255.1:39689)
0926 05:54:03.821916 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:03.821928 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:03.821988 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:03.822237 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:03.822239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:03.822241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:03.822310 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.822311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.823116 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.823117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.823976 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:03.823979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.824781 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.824782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:03.824859 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.824864 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:54:03.824913 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 05:54:03.824955 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"thread_start_us":146,"threads_started":1}
I20260926 05:54:03.825038 7369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.821068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:39689 (local address 127.24.255.3:45331)
0926 05:54:03.821255 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:03.821258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:03.821980 (+ 722us) server_negotiation.cc:407] Connection header received
0926 05:54:03.822084 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:03.822085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:03.822121 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:03.822184 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:03.822385 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.823055 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.824066 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.824587 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.824761 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.824773 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:03.824929 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 05:54:03.824980 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:54:03.825289 7309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d484755f550f410c8f0f19176f49676d" candidate_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" is_pre_election: true
I20260926 05:54:03.825390 7309 raft_consensus.cc:2454] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe6f885c67b14c259ba22f0b5d70c0bb in term 0.
I20260926 05:54:03.825680 7378 raft_consensus.cc:504] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } } peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } }
I20260926 05:54:03.825799 7378 leader_election.cc:288] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [CANDIDATE]: Term 1 election: Requested vote from peers feb45481d05e4dcfa6cc77e4d5bf5f30 (127.24.255.3:45331), b06214c9867b494c84313036fed37dcf (127.24.255.2:44457)
I20260926 05:54:03.826007 7309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d484755f550f410c8f0f19176f49676d" candidate_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30"
I20260926 05:54:03.825991 7175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d484755f550f410c8f0f19176f49676d" candidate_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06214c9867b494c84313036fed37dcf"
I20260926 05:54:03.826084 7309 raft_consensus.cc:3037] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:03.826085 7175 raft_consensus.cc:3037] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:03.827199 7175 raft_consensus.cc:2454] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe6f885c67b14c259ba22f0b5d70c0bb in term 1.
I20260926 05:54:03.827200 7309 raft_consensus.cc:2454] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe6f885c67b14c259ba22f0b5d70c0bb in term 1.
I20260926 05:54:03.827378 6974 leader_election.cc:302] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fe6f885c67b14c259ba22f0b5d70c0bb, feb45481d05e4dcfa6cc77e4d5bf5f30; no voters:
I20260926 05:54:03.827613 7378 raft_consensus.cc:2781] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:54:03.827870 7378 raft_consensus.cc:686] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 LEADER]: Becoming Leader. State: Replica: fe6f885c67b14c259ba22f0b5d70c0bb, State: Running, Role: LEADER
I20260926 05:54:03.827986 7378 consensus_queue.cc:237] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } } peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } }
I20260926 05:54:03.829279 6896 catalog_manager.cc:5495] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb reported cstate change: term changed from 0 to 1, leader changed from <none> to fe6f885c67b14c259ba22f0b5d70c0bb (127.24.255.1). New cstate: current_term: 1 leader_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } health_report { overall_health: UNKNOWN } } }
I20260926 05:54:03.836621 7368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.833627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.24.255.1:38011)
0926 05:54:03.833712 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:03.833713 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:03.833722 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:54:03.833754 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:03.833755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:03.833774 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:03.833845 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:54:03.834046 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.834614 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.835392 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:03.835639 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:03.835695 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:03.835992 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:54:03.836152 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:54:03.836284 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:03.836506 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 05:54:03.836531 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 05:54:03.838677 7309 raft_consensus.cc:1261] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 1 FOLLOWER]: Refusing update from remote peer fe6f885c67b14c259ba22f0b5d70c0bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:54:03.838687 7175 raft_consensus.cc:1261] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Refusing update from remote peer fe6f885c67b14c259ba22f0b5d70c0bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:54:03.839154 7382 consensus_queue.cc:1035] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:54:03.839318 7378 consensus_queue.cc:1035] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:54:03.841812 7385 mvcc.cc:205] Tried to move back new op lower bound from 7333486771559669760 to 7333486771520684032. Current Snapshot: MvccSnapshot[applied={T|T < 7333486771559669760}]
I20260926 05:54:03.844301 6894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:54:03.844403 6894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:54:03.845257 6894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Sending DeleteTablet for 3 replicas of tablet d484755f550f410c8f0f19176f49676d
I20260926 05:54:03.845602 7021 tablet_service.cc:1514] Processing DeleteTablet for tablet d484755f550f410c8f0f19176f49676d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:54:03 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:59692
I20260926 05:54:03.845705 7155 tablet_service.cc:1514] Processing DeleteTablet for tablet d484755f550f410c8f0f19176f49676d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:54:03 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:43578
I20260926 05:54:03.845860 7395 tablet_replica.cc:330] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: stopping tablet replica
I20260926 05:54:03.845949 7395 raft_consensus.cc:2229] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:54:03.845921 7396 tablet_replica.cc:330] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf: stopping tablet replica
I20260926 05:54:03.846019 7289 tablet_service.cc:1514] Processing DeleteTablet for tablet d484755f550f410c8f0f19176f49676d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:54:03 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:57722
I20260926 05:54:03.846071 7396 raft_consensus.cc:2229] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:54:03.846088 7395 raft_consensus.cc:2258] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:54:03.846203 7396 raft_consensus.cc:2258] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:54:03.846388 7397 tablet_replica.cc:330] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30: stopping tablet replica
I20260926 05:54:03.846459 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 6958
I20260926 05:54:03.846504 7397 raft_consensus.cc:2229] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:54:03.846603 7397 raft_consensus.cc:2258] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:54:03.847736 7397 ts_tablet_manager.cc:1905] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:54:03.848170 7396 ts_tablet_manager.cc:1905] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:54:03.849598 7396 ts_tablet_manager.cc:1918] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:54:03.849685 7396 log.cc:1196] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/d484755f550f410c8f0f19176f49676d
I20260926 05:54:03.849803 7397 ts_tablet_manager.cc:1918] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:54:03.849951 7396 ts_tablet_manager.cc:1939] T d484755f550f410c8f0f19176f49676d P b06214c9867b494c84313036fed37dcf: Deleting consensus metadata
I20260926 05:54:03.850028 7397 log.cc:1196] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/d484755f550f410c8f0f19176f49676d
I20260926 05:54:03.850642 7397 ts_tablet_manager.cc:1939] T d484755f550f410c8f0f19176f49676d P feb45481d05e4dcfa6cc77e4d5bf5f30: Deleting consensus metadata
I20260926 05:54:03.850608 6883 catalog_manager.cc:4841] TS b06214c9867b494c84313036fed37dcf (127.24.255.2:44457): tablet d484755f550f410c8f0f19176f49676d (table test-table [id=bbb687dd7c674708af6c070d9a667956]) successfully deleted
I20260926 05:54:03.851042 6880 catalog_manager.cc:4841] TS feb45481d05e4dcfa6cc77e4d5bf5f30 (127.24.255.3:45331): tablet d484755f550f410c8f0f19176f49676d (table test-table [id=bbb687dd7c674708af6c070d9a667956]) successfully deleted
W20260926 05:54:03.852807 6883 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:38011 (error 108)
I20260926 05:54:03.852847 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 7092
I20260926 05:54:03.855221 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:03.855089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:38011 (local address 127.0.0.1:59698)
0926 05:54:03.855142 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:03.855184 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:38011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 05:54:03.855384 6883 catalog_manager.cc:4584] TS fe6f885c67b14c259ba22f0b5d70c0bb (127.24.255.1:38011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d484755f550f410c8f0f19176f49676d: Network error: Client connection negotiation failed: client connection to 127.24.255.1:38011: connect: Connection refused (error 111)
I20260926 05:54:03.858045 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 7226
I20260926 05:54:03.862047 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 6864
I20260926 05:54:03.867017 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:35393
--webserver_interface=127.24.255.62
--webserver_port=44755
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:35393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:54:03.959044 7399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:03.959282 7399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:03.959308 7399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:03.960654 7399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:54:03.960721 7399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:03.960734 7399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:03.960747 7399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:54:03.960758 7399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:54:03.962581 7399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:35393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:35393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=44755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:03.963229 7399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:03.963496 7399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:03.975106 7406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:03.975095 7408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:03.975558 7405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:03.975718 7399 server_base.cc:1034] running on GCE node
I20260926 05:54:03.975998 7399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:03.976699 7399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:03.977946 7399 hybrid_clock.cc:648] HybridClock initialized: now 1790402043977919 us; error 37 us; skew 500 ppm
Sep 26 05:54:03 dist-test-slave-7fr8 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402043, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:03.981115 7399 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:54:03.981616 7399 webserver.cc:460] Webserver started at http://127.24.255.62:44755/ using document root <none> and password file <none>
I20260926 05:54:03.981859 7399 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:03.981926 7399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:03.983588 7399 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:03.984637 7415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:03.985090 7399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:03.985206 7399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "8b8f701658e040b8a114896b4175b52a"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "348ac58b300a568c8d8e901d424b2711"
server_key_iv: "755095b338d098f0ee21e4ee584f1f27"
server_key_version: "encryptionkey@0"
I20260926 05:54:03.985736 7399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:03.993767 7399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:03.996096 7399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:03.996249 7399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:04.000186 7399 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:35393
I20260926 05:54:04.000253 7467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:35393 every 8 connection(s)
I20260926 05:54:04.000619 7399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:54:04.002369 7468 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:54:04.003062 7468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Bootstrap starting.
I20260926 05:54:04.003050 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 7399
I20260926 05:54:04.003450 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:38011
--local_ip_for_outbound_sockets=127.24.255.1
--tserver_master_addrs=127.24.255.62:35393
--webserver_port=35959
--webserver_interface=127.24.255.1
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:54:04.011626 7468 log.cc:824] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:04.014336 7468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:54:04.014632 7468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Bootstrap complete.
I20260926 05:54:04.016782 7468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } }
I20260926 05:54:04.017033 7468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8f701658e040b8a114896b4175b52a, State: Initialized, Role: FOLLOWER
I20260926 05:54:04.017175 7468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } }
I20260926 05:54:04.017238 7468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:54:04.017284 7468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:54:04.017325 7468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:54:04.018462 7468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } }
I20260926 05:54:04.018554 7468 leader_election.cc:302] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b8f701658e040b8a114896b4175b52a; no voters:
I20260926 05:54:04.018718 7468 leader_election.cc:288] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 05:54:04.018777 7472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:54:04.018890 7472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 2 LEADER]: Becoming Leader. State: Replica: 8b8f701658e040b8a114896b4175b52a, State: Running, Role: LEADER
I20260926 05:54:04.019001 7468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:54:04.019043 7472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } }
I20260926 05:54:04.019546 7472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b8f701658e040b8a114896b4175b52a. Latest consensus state: current_term: 2 leader_uuid: "8b8f701658e040b8a114896b4175b52a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } } }
I20260926 05:54:04.019616 7474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8b8f701658e040b8a114896b4175b52a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } } }
I20260926 05:54:04.019723 7472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: This master's current role is: LEADER
I20260926 05:54:04.019737 7474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: This master's current role is: LEADER
I20260926 05:54:04.020092 7481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:54:04.021122 7481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:54:04.021230 7481 catalog_manager.cc:1259] Loaded cluster ID: 610c8d3c949e4198bdab12aa34776db1
I20260926 05:54:04.021296 7481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:54:04.021567 7481 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 05:54:04.026463 7481 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 05:54:04.029225 7481 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:54:04.029490 7481 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Loaded TSK: 0
I20260926 05:54:04.029731 7481 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 05:54:04.103549 7470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:04.103788 7470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:04.103828 7470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:04.105201 7470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:04.105257 7470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:04.105301 7470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:54:04.107038 7470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:38011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=35959
--tserver_master_addrs=127.24.255.62:35393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:04.107458 7470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:04.107745 7470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:04.113209 7498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:04.113197 7496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:04.113232 7495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:04.113623 7470 server_base.cc:1034] running on GCE node
I20260926 05:54:04.113847 7470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:04.114154 7470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:04.115334 7470 hybrid_clock.cc:648] HybridClock initialized: now 1790402044115314 us; error 33 us; skew 500 ppm
Sep 26 05:54:04 dist-test-slave-7fr8 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402044, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:04.118211 7470 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:54:04.118685 7470 webserver.cc:460] Webserver started at http://127.24.255.1:35959/ using document root <none> and password file <none>
I20260926 05:54:04.118920 7470 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:04.118988 7470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:04.121069 7470 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:04.122313 7505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:04.122543 7470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 05:54:04.122677 7470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "fe6f885c67b14c259ba22f0b5d70c0bb"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "ee91f9207c24bbdedb3d2097e3a29985"
server_key_iv: "826f13fe95715f8b636acfeaecb502c2"
server_key_version: "encryptionkey@0"
I20260926 05:54:04.123190 7470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:04.162362 7470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:04.165289 7470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:04.165452 7470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:04.165726 7470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:04.166590 7512 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 05:54:04.168751 7470 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 05:54:04.168825 7470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260926 05:54:04.168864 7470 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 05:54:04.169582 7470 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 05:54:04.169617 7470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 05:54:04.169883 7512 tablet_bootstrap.cc:492] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Bootstrap starting.
I20260926 05:54:04.171844 7512 log.cc:824] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:04.175009 7512 tablet_bootstrap.cc:492] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:54:04.175313 7512 tablet_bootstrap.cc:492] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Bootstrap complete.
I20260926 05:54:04.175450 7512 ts_tablet_manager.cc:1397] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 05:54:04.176947 7470 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:38011
I20260926 05:54:04.176977 7620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:38011 every 8 connection(s)
I20260926 05:54:04.177265 7470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:54:04.177290 7512 raft_consensus.cc:348] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } } peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } }
I20260926 05:54:04.178032 7512 raft_consensus.cc:729] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6f885c67b14c259ba22f0b5d70c0bb, State: Initialized, Role: FOLLOWER
I20260926 05:54:04.178241 7512 consensus_queue.cc:260] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } } peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } }
I20260926 05:54:04.178494 7512 ts_tablet_manager.cc:1428] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:54:04.181373 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 7470
I20260926 05:54:04.181697 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:44457
--local_ip_for_outbound_sockets=127.24.255.2
--tserver_master_addrs=127.24.255.62:35393
--webserver_port=33267
--webserver_interface=127.24.255.2
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:54:04 dist-test-slave-7fr8 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402044, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:54:04.189011 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.178029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:47507 (local address 127.24.255.62:35393)
0926 05:54:04.179018 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:04.179024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:04.179048 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:54:04.179152 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:04.179156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:04.179193 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:04.179303 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:04.179941 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.180818 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.182205 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.182838 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.182965 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.185531 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:04.185536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:04.185536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:04.185548 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:04.187298 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:04.187710 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:04.187713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:04.187714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:04.187809 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:04.188030 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:04.188031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:04.188031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:04.188296 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:04.188369 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:04.188680 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 05:54:04.188806 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":110,"threads_started":1}
I20260926 05:54:04.189023 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.177934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:47507)
0926 05:54:04.178675 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:04.178708 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:04.178834 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:04.179348 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:04.179352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:04.179629 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:04.179820 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.179825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.180926 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.180927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.182129 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.182132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.182950 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.182951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.183121 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.183652 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:04.183661 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:04.185413 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:04.187388 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:04.187396 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:04.187398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:04.187634 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:04.187868 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:04.187869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:04.187869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:04.187973 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:04.188359 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:04.188366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:04.188501 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 05:54:04.188595 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 05:54:04.188786 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":75,"threads_started":1}
I20260926 05:54:04.190133 7621 heartbeater.cc:347] Connected to a master server at 127.24.255.62:35393
I20260926 05:54:04.190234 7621 heartbeater.cc:464] Registering TS with master...
I20260926 05:54:04.190409 7621 heartbeater.cc:511] Master 127.24.255.62:35393 requested a full tablet report, sending...
I20260926 05:54:04.191133 7432 ts_manager.cc:194] Registered new tserver with Master: fe6f885c67b14c259ba22f0b5d70c0bb (127.24.255.1:38011)
I20260926 05:54:04.192162 7432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:47507
I20260926 05:54:04.193068 7621 heartbeater.cc:503] Master 127.24.255.62:35393 was elected leader, sending a full tablet report...
I20260926 05:54:04.198087 7628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.191805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.24.255.1:38011)
0926 05:54:04.191963 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:04.191967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:04.194355 (+ 2388us) server_negotiation.cc:407] Connection header received
0926 05:54:04.194453 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:04.194459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:04.194474 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:04.194534 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:04.194679 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.195358 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.196877 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.197492 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.197563 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.197614 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:04.197937 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 05:54:04.197991 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":100,"threads_started":1}
I20260926 05:54:04.198086 7627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.191708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:38011 (local address 127.0.0.1:59700)
0926 05:54:04.194220 (+ 2512us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:04.194266 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:04.194349 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:04.194555 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:04.194558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:04.194560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:04.194629 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.194631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.195426 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.195427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.196770 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.196774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.197678 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.197680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.197786 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.197802 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 05:54:04.197885 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 05:54:04.197995 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2475,"thread_start_us":100,"threads_started":1}
I20260926 05:54:04.198933 7555 tablet_service.cc:1514] Processing DeleteTablet for tablet d484755f550f410c8f0f19176f49676d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:54:03 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:59700
I20260926 05:54:04.199841 7629 tablet_replica.cc:330] stopping tablet replica
I20260926 05:54:04.200012 7629 raft_consensus.cc:2229] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:54:04.200105 7629 raft_consensus.cc:2258] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:54:04.200522 7629 ts_tablet_manager.cc:1905] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:54:04.204952 7629 ts_tablet_manager.cc:1918] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:54:04.205200 7629 log.cc:1196] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal/wals/d484755f550f410c8f0f19176f49676d
I20260926 05:54:04.205576 7629 ts_tablet_manager.cc:1939] T d484755f550f410c8f0f19176f49676d P fe6f885c67b14c259ba22f0b5d70c0bb: Deleting consensus metadata
I20260926 05:54:04.206511 7419 catalog_manager.cc:4841] TS fe6f885c67b14c259ba22f0b5d70c0bb (127.24.255.1:38011): tablet d484755f550f410c8f0f19176f49676d (table test-table [id=bbb687dd7c674708af6c070d9a667956]) successfully deleted
W20260926 05:54:04.276023 7626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:04.276271 7626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:04.276298 7626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:04.277541 7626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:04.277593 7626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:04.277621 7626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:54:04.279194 7626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:44457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=33267
--tserver_master_addrs=127.24.255.62:35393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:04.279435 7626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:04.279673 7626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:04.284797 7635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:04.284971 7637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:04.285054 7626 server_base.cc:1034] running on GCE node
W20260926 05:54:04.284811 7634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:04.285375 7626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:04.285934 7626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:04.287187 7626 hybrid_clock.cc:648] HybridClock initialized: now 1790402044287149 us; error 51 us; skew 500 ppm
Sep 26 05:54:04 dist-test-slave-7fr8 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402044, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:04.294806 7626 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:54:04.295320 7626 webserver.cc:460] Webserver started at http://127.24.255.2:33267/ using document root <none> and password file <none>
I20260926 05:54:04.295531 7626 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:04.295589 7626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:04.297031 7626 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:04.297663 7644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:04.297842 7626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:04.297958 7626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "b06214c9867b494c84313036fed37dcf"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "b7054c158ad0f29ee5fa5bc964ae3aae"
server_key_iv: "ddc912c8a8aab03b865f35e74a21cd2f"
server_key_version: "encryptionkey@0"
I20260926 05:54:04.298357 7626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:04.320523 7626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:04.322716 7626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:04.322860 7626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:04.323086 7626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:04.323378 7626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:54:04.323411 7626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:04.323441 7626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:54:04.323474 7626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:04.329058 7626 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:44457
I20260926 05:54:04.329103 7757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:44457 every 8 connection(s)
I20260926 05:54:04.329445 7626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:54:04 dist-test-slave-7fr8 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402044, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:54:04.338893 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 7626
I20260926 05:54:04.339196 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:45331
--local_ip_for_outbound_sockets=127.24.255.3
--tserver_master_addrs=127.24.255.62:35393
--webserver_port=36499
--webserver_interface=127.24.255.3
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:54:04.339362 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.330267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:46717 (local address 127.24.255.62:35393)
0926 05:54:04.330357 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:04.330358 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:04.330586 (+ 228us) server_negotiation.cc:407] Connection header received
0926 05:54:04.331153 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:04.331154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:04.331166 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:04.331229 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:04.332259 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.332752 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.333925 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.334188 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.334254 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.336519 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:04.336521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:04.336521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:04.336529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:04.337839 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:04.338355 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:04.338356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:04.338356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:04.338409 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:04.338684 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:04.338684 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:04.338685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:04.338789 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:04.338833 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:04.339175 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 05:54:04.339256 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 05:54:04.339715 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.330269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:46717)
0926 05:54:04.330432 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:04.330476 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:04.331012 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:04.331283 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:04.331286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:04.331884 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:04.332088 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.332093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.332867 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.332868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.333817 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.333819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.334264 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.334265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.334373 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.334939 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:04.334948 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:04.336379 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:04.337979 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:04.337986 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:04.337987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:04.338286 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:04.338492 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:04.338493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:04.338493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:04.338625 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:04.338847 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:04.338852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:04.339038 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 05:54:04.339161 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 05:54:04.339344 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260926 05:54:04.340305 7758 heartbeater.cc:347] Connected to a master server at 127.24.255.62:35393
I20260926 05:54:04.340402 7758 heartbeater.cc:464] Registering TS with master...
I20260926 05:54:04.340590 7758 heartbeater.cc:511] Master 127.24.255.62:35393 requested a full tablet report, sending...
I20260926 05:54:04.340977 7432 ts_manager.cc:194] Registered new tserver with Master: b06214c9867b494c84313036fed37dcf (127.24.255.2:44457)
I20260926 05:54:04.341485 7432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:46717
W20260926 05:54:04.421482 7761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:04.421711 7761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:04.421737 7761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:04.423523 7761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:04.423595 7761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:04.423627 7761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:54:04.425732 7761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:45331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=36499
--tserver_master_addrs=127.24.255.62:35393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:04.426024 7761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:04.426570 7761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:04.432241 7766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:04.432313 7761 server_base.cc:1034] running on GCE node
W20260926 05:54:04.432276 7767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:04.432258 7769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:04.432744 7761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:04.433023 7761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:04.434267 7761 hybrid_clock.cc:648] HybridClock initialized: now 1790402044434240 us; error 45 us; skew 500 ppm
Sep 26 05:54:04 dist-test-slave-7fr8 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402044, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:04.437057 7761 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:54:04.437554 7761 webserver.cc:460] Webserver started at http://127.24.255.3:36499/ using document root <none> and password file <none>
I20260926 05:54:04.437757 7761 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:04.437822 7761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:04.439254 7761 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:04.440125 7776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:04.440451 7761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:04.440562 7761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "6dc9e846bba4ad644cafd68543b48882"
server_key_iv: "88b8aeb119e517c27e70cbc668582dbf"
server_key_version: "encryptionkey@0"
I20260926 05:54:04.440930 7761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:04.453550 7761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:04.455875 7761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:04.456019 7761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:04.456259 7761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:04.456570 7761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:54:04.456606 7761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:04.456638 7761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:54:04.456656 7761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:04.462296 7761 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:45331
I20260926 05:54:04.462344 7889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:45331 every 8 connection(s)
I20260926 05:54:04.462695 7761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:54:04.464967 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 7761
Sep 26 05:54:04 dist-test-slave-7fr8 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402044, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:54:04.473129 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.465356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.24.255.62:35393)
0926 05:54:04.465459 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:04.465461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:04.465608 (+ 147us) server_negotiation.cc:407] Connection header received
0926 05:54:04.465682 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:04.465683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:04.465694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:04.465752 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:04.466367 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.466862 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.467600 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.467791 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.467857 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.469188 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:04.469189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:04.469190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:04.469197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:04.471234 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:04.471859 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:04.471862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:04.471862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:04.472050 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:04.472391 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:04.472393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:04.472393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:04.472594 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:04.472856 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:04.472931 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 05:54:04.473019 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":84,"thread_start_us":54,"threads_started":1}
I20260926 05:54:04.473258 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.463375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:59377 (local address 127.24.255.62:35393)
0926 05:54:04.463418 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:04.463420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:04.463619 (+ 199us) server_negotiation.cc:407] Connection header received
0926 05:54:04.464090 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:04.464091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:04.464102 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:04.464150 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:04.465096 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.465710 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.466730 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.466938 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.466995 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.469499 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:04.469501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:04.469501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:04.469506 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:04.471186 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:04.471859 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:04.471862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:04.471862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:04.472038 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:04.472454 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:04.472455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:04.472456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:04.472594 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:04.472746 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:04.473116 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 05:54:04.473185 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":18}
I20260926 05:54:04.473519 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.463365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:59377)
0926 05:54:04.463502 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:04.463539 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:04.464010 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:04.464205 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:04.464208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:04.464775 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:04.464974 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.464979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.465802 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.465803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.466641 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.466644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.467023 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.467024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.467151 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.467680 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:04.467689 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:04.469404 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:04.471317 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:04.471329 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:04.471332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:04.471684 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:04.472155 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:04.472155 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:04.472156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:04.472326 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:04.472749 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:04.472757 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:04.472946 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 05:54:04.473097 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 05:54:04.473286 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260926 05:54:04.473990 7890 heartbeater.cc:347] Connected to a master server at 127.24.255.62:35393
I20260926 05:54:04.474114 7890 heartbeater.cc:464] Registering TS with master...
I20260926 05:54:04.474300 7890 heartbeater.cc:511] Master 127.24.255.62:35393 requested a full tablet report, sending...
I20260926 05:54:04.474632 7432 ts_manager.cc:194] Registered new tserver with Master: feb45481d05e4dcfa6cc77e4d5bf5f30 (127.24.255.3:45331)
I20260926 05:54:04.475014 7432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:59377
I20260926 05:54:04.485877 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:54:04.489235 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.486540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.24.255.62:35393)
0926 05:54:04.486656 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:04.486659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:04.486749 (+ 90us) server_negotiation.cc:407] Connection header received
0926 05:54:04.486846 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:04.486846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:04.486891 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:04.486970 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:54:04.487115 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.487659 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.488512 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.488717 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.488774 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.488925 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:54:04.489032 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:54:04.489113 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:04.489129 (+ 16us) server_negotiation.cc:299] Negotiation successful
0926 05:54:04.489153 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 05:54:04.489573 7432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:54:04.491385 7432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:54:04.494217 7555 tablet_service.cc:1467] Processing CreateTablet for tablet 4184972c588b4621b1c2d4d35ccf2511 (DEFAULT_TABLE table=test-table [id=20cdd920127145eea7b9535e7e70ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:54:04.494431 7555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4184972c588b4621b1c2d4d35ccf2511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:04.495976 7512 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb: Bootstrap starting.
I20260926 05:54:04.496402 7512 tablet_bootstrap.cc:654] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:04.497018 7512 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb: No bootstrap required, opened a new log
I20260926 05:54:04.497072 7512 ts_tablet_manager.cc:1397] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:04.497208 7512 raft_consensus.cc:348] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:04.497301 7512 raft_consensus.cc:374] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:04.497351 7512 raft_consensus.cc:729] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6f885c67b14c259ba22f0b5d70c0bb, State: Initialized, Role: FOLLOWER
I20260926 05:54:04.497395 7512 consensus_queue.cc:260] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:04.497483 7512 ts_tablet_manager.cc:1428] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:54:04.497716 7627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.493655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:45331 (local address 127.0.0.1:57730)
0926 05:54:04.493708 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:04.493721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:04.493768 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:04.494253 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:04.494254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:04.494256 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:04.494295 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.494296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.495289 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.495290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.496531 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.496534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.497458 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.497459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.497533 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.497540 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 05:54:04.497586 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:54:04.497624 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":26}
I20260926 05:54:04.497740 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.493847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.24.255.3:45331)
0926 05:54:04.494034 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:04.494039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:04.494050 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:54:04.494091 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:04.494093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:04.494133 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:04.494361 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:04.494411 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.495191 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.496633 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.497384 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.497459 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.497526 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:04.497615 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 05:54:04.497660 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260926 05:54:04.497716 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.493656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:44457 (local address 127.0.0.1:43586)
0926 05:54:04.493807 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:04.493817 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:04.493859 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:04.494217 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:04.494218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:04.494220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:04.494276 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.494277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.495487 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.495488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.496600 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.496601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.497458 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.497459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.497533 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.497540 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 05:54:04.497569 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 05:54:04.497620 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":28,"thread_start_us":78,"threads_started":1}
I20260926 05:54:04.498168 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.493848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.24.255.2:44457)
0926 05:54:04.494039 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:04.494043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:04.494050 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:54:04.494091 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:04.494114 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:04.494131 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:04.494219 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:04.494365 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.495384 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.496686 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.497363 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.497753 (+ 390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.497822 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:04.497849 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 05:54:04.498043 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
I20260926 05:54:04.498672 7824 tablet_service.cc:1467] Processing CreateTablet for tablet 4184972c588b4621b1c2d4d35ccf2511 (DEFAULT_TABLE table=test-table [id=20cdd920127145eea7b9535e7e70ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:54:04.499004 7824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4184972c588b4621b1c2d4d35ccf2511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:04.501974 7900 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Bootstrap starting.
I20260926 05:54:04.502364 7692 tablet_service.cc:1467] Processing CreateTablet for tablet 4184972c588b4621b1c2d4d35ccf2511 (DEFAULT_TABLE table=test-table [id=20cdd920127145eea7b9535e7e70ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:54:04.502719 7692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4184972c588b4621b1c2d4d35ccf2511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:04.503144 7900 tablet_bootstrap.cc:654] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:04.503839 7900 log.cc:824] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:04.504981 7900 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: No bootstrap required, opened a new log
I20260926 05:54:04.505110 7900 ts_tablet_manager.cc:1397] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:54:04.505600 7902 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: Bootstrap starting.
I20260926 05:54:04.506278 7902 tablet_bootstrap.cc:654] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:04.506722 7902 log.cc:824] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:04.506723 7900 raft_consensus.cc:348] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:04.506879 7900 raft_consensus.cc:374] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:04.506906 7900 raft_consensus.cc:729] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb45481d05e4dcfa6cc77e4d5bf5f30, State: Initialized, Role: FOLLOWER
I20260926 05:54:04.507128 7900 consensus_queue.cc:260] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:04.507409 7902 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: No bootstrap required, opened a new log
I20260926 05:54:04.507467 7900 ts_tablet_manager.cc:1428] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:54:04.507498 7890 heartbeater.cc:503] Master 127.24.255.62:35393 was elected leader, sending a full tablet report...
I20260926 05:54:04.507511 7902 ts_tablet_manager.cc:1397] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:54:04.509012 7902 raft_consensus.cc:348] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:04.509163 7902 raft_consensus.cc:374] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:04.509197 7902 raft_consensus.cc:729] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06214c9867b494c84313036fed37dcf, State: Initialized, Role: FOLLOWER
I20260926 05:54:04.509310 7902 consensus_queue.cc:260] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:04.509624 7902 ts_tablet_manager.cc:1428] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:54:04.509637 7758 heartbeater.cc:503] Master 127.24.255.62:35393 was elected leader, sending a full tablet report...
I20260926 05:54:04.550377 7904 raft_consensus.cc:482] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:54:04.550516 7904 raft_consensus.cc:504] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:04.551014 7904 leader_election.cc:288] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b06214c9867b494c84313036fed37dcf (127.24.255.2:44457), fe6f885c67b14c259ba22f0b5d70c0bb (127.24.255.1:38011)
I20260926 05:54:04.555550 7628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.551527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:60891 (local address 127.24.255.1:38011)
0926 05:54:04.551735 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:04.551738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:04.551747 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:54:04.551774 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:04.551775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:04.551789 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:04.551877 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:04.552109 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.552795 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.554276 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.555116 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.555208 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.555220 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:04.555413 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 05:54:04.555468 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260926 05:54:04.555565 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.551283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:38011 (local address 127.24.255.3:60891)
0926 05:54:04.551604 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:04.551614 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:04.551651 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:04.551948 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:04.551949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:04.551951 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:04.552021 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.552022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.552879 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.552880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.553990 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.553992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.555227 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.555229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.555322 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.555327 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:54:04.555392 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 05:54:04.555444 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":6,"thread_start_us":103,"threads_started":1}
I20260926 05:54:04.555604 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.551343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:40247 (local address 127.24.255.2:44457)
0926 05:54:04.551419 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:04.551421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:04.551431 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:54:04.551454 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:04.551455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:04.551466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:04.551511 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:04.551757 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.552591 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.554219 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.555143 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.555216 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.555225 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:04.555413 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 05:54:04.555468 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 05:54:04.555647 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.551168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:44457 (local address 127.24.255.3:40247)
0926 05:54:04.551223 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:04.551236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:04.551281 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:04.551527 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:04.551528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:04.551530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:04.551601 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.551603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.552706 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.552707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.553965 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:04.553968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.555228 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.555229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:04.555326 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.555328 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:54:04.555392 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:54:04.555445 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":13}
I20260926 05:54:04.555987 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4184972c588b4621b1c2d4d35ccf2511" candidate_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06214c9867b494c84313036fed37dcf" is_pre_election: true
I20260926 05:54:04.556150 7712 raft_consensus.cc:2454] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb45481d05e4dcfa6cc77e4d5bf5f30 in term 0.
I20260926 05:54:04.556142 7575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4184972c588b4621b1c2d4d35ccf2511" candidate_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" is_pre_election: true
I20260926 05:54:04.556334 7575 raft_consensus.cc:2454] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb45481d05e4dcfa6cc77e4d5bf5f30 in term 0.
I20260926 05:54:04.556423 7780 leader_election.cc:302] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b06214c9867b494c84313036fed37dcf, feb45481d05e4dcfa6cc77e4d5bf5f30; no voters:
I20260926 05:54:04.556574 7904 raft_consensus.cc:2781] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:54:04.556658 7904 raft_consensus.cc:482] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:54:04.556699 7904 raft_consensus.cc:3037] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:04.557436 7904 raft_consensus.cc:504] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:04.557550 7904 leader_election.cc:288] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [CANDIDATE]: Term 1 election: Requested vote from peers b06214c9867b494c84313036fed37dcf (127.24.255.2:44457), fe6f885c67b14c259ba22f0b5d70c0bb (127.24.255.1:38011)
I20260926 05:54:04.557773 7712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4184972c588b4621b1c2d4d35ccf2511" candidate_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06214c9867b494c84313036fed37dcf"
I20260926 05:54:04.557768 7575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4184972c588b4621b1c2d4d35ccf2511" candidate_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb"
I20260926 05:54:04.557847 7712 raft_consensus.cc:3037] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:04.557848 7575 raft_consensus.cc:3037] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:04.558588 7712 raft_consensus.cc:2454] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb45481d05e4dcfa6cc77e4d5bf5f30 in term 1.
I20260926 05:54:04.558658 7575 raft_consensus.cc:2454] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb45481d05e4dcfa6cc77e4d5bf5f30 in term 1.
I20260926 05:54:04.558799 7780 leader_election.cc:302] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fe6f885c67b14c259ba22f0b5d70c0bb, feb45481d05e4dcfa6cc77e4d5bf5f30; no voters:
I20260926 05:54:04.558945 7904 raft_consensus.cc:2781] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:54:04.559103 7904 raft_consensus.cc:686] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 1 LEADER]: Becoming Leader. State: Replica: feb45481d05e4dcfa6cc77e4d5bf5f30, State: Running, Role: LEADER
I20260926 05:54:04.559212 7904 consensus_queue.cc:237] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:04.560134 7432 catalog_manager.cc:5495] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb45481d05e4dcfa6cc77e4d5bf5f30 (127.24.255.3). New cstate: current_term: 1 leader_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } health_report { overall_health: UNKNOWN } } }
I20260926 05:54:04.576542 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.574274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.24.255.3:45331)
0926 05:54:04.574338 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:04.574340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:04.574348 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:54:04.574434 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:04.574435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:04.574453 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:04.574512 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:54:04.574627 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.575113 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.575830 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:04.576028 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:04.576082 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:04.576214 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:54:04.576320 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:54:04.576401 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:04.576416 (+ 15us) server_negotiation.cc:299] Negotiation successful
0926 05:54:04.576439 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:54:04.578976 7575 raft_consensus.cc:1261] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 FOLLOWER]: Refusing update from remote peer feb45481d05e4dcfa6cc77e4d5bf5f30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:54:04.578975 7712 raft_consensus.cc:1261] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Refusing update from remote peer feb45481d05e4dcfa6cc77e4d5bf5f30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:54:04.579319 7904 consensus_queue.cc:1035] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:54:04.579478 7907 consensus_queue.cc:1035] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 05:54:04.580036 7759 tablet.cc:2369] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:54:04.582015 7912 mvcc.cc:205] Tried to move back new op lower bound from 7333486774591475712 to 7333486774515253248. Current Snapshot: MvccSnapshot[applied={T|T < 7333486774591475712}]
I20260926 05:54:04.582868 7911 mvcc.cc:205] Tried to move back new op lower bound from 7333486774591475712 to 7333486774515253248. Current Snapshot: MvccSnapshot[applied={T|T < 7333486774591475712 or (T in {7333486774591475712})}]
I20260926 05:54:04.585022 7430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:54:04.585161 7430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:54:04.585997 7430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Sending DeleteTablet for 3 replicas of tablet 4184972c588b4621b1c2d4d35ccf2511
I20260926 05:54:04.586396 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 7470
I20260926 05:54:04.586405 7692 tablet_service.cc:1514] Processing DeleteTablet for tablet 4184972c588b4621b1c2d4d35ccf2511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:54:04 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:43586
I20260926 05:54:04.588900 7824 tablet_service.cc:1514] Processing DeleteTablet for tablet 4184972c588b4621b1c2d4d35ccf2511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:54:04 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:57730
I20260926 05:54:04.589200 7921 tablet_replica.cc:330] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: stopping tablet replica
I20260926 05:54:04.589323 7921 raft_consensus.cc:2229] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:54:04.589512 7921 raft_consensus.cc:2258] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:54:04.589984 7921 ts_tablet_manager.cc:1905] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:54:04.592070 7921 ts_tablet_manager.cc:1918] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:54:04.592154 7921 log.cc:1196] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/4184972c588b4621b1c2d4d35ccf2511
I20260926 05:54:04.592454 7921 ts_tablet_manager.cc:1939] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Deleting consensus metadata
W20260926 05:54:04.592824 7419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:38011 (error 108)
I20260926 05:54:04.593040 7920 tablet_replica.cc:330] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: stopping tablet replica
I20260926 05:54:04.593163 7920 raft_consensus.cc:2229] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:54:04.593290 7920 raft_consensus.cc:2258] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:54:04.593401 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.593277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:38011 (local address 127.0.0.1:59716)
0926 05:54:04.593333 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:04.593370 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:38011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 05:54:04.593513 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 7626
W20260926 05:54:04.593545 7419 catalog_manager.cc:4584] TS fe6f885c67b14c259ba22f0b5d70c0bb (127.24.255.1:38011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4184972c588b4621b1c2d4d35ccf2511: Network error: Client connection negotiation failed: client connection to 127.24.255.1:38011: connect: Connection refused (error 111)
I20260926 05:54:04.598362 7416 catalog_manager.cc:4841] TS feb45481d05e4dcfa6cc77e4d5bf5f30 (127.24.255.3:45331): tablet 4184972c588b4621b1c2d4d35ccf2511 (table test-table [id=20cdd920127145eea7b9535e7e70ed02]) successfully deleted
I20260926 05:54:04.598558 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 7761
I20260926 05:54:04.598623 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.598484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:44457 (local address 127.0.0.1:43598)
0926 05:54:04.598557 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:04.598582 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.2:44457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 05:54:04.603113 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 7399
I20260926 05:54:04.608248 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:35393
--webserver_interface=127.24.255.62
--webserver_port=44755
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:35393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:54:04.691605 7923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:04.691931 7923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:04.691962 7923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:04.694270 7923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:54:04.694352 7923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:04.694368 7923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:04.694382 7923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:54:04.694396 7923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:54:04.696657 7923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:35393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:35393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=44755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:04.696957 7923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:04.697216 7923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:04.702778 7931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:04.702785 7928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:04.702971 7923 server_base.cc:1034] running on GCE node
W20260926 05:54:04.702778 7929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:04.703639 7923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:04.704013 7923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:04.705204 7923 hybrid_clock.cc:648] HybridClock initialized: now 1790402044705149 us; error 69 us; skew 500 ppm
Sep 26 05:54:04 dist-test-slave-7fr8 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402044, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:04.708585 7923 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:54:04.709240 7923 webserver.cc:460] Webserver started at http://127.24.255.62:44755/ using document root <none> and password file <none>
I20260926 05:54:04.709470 7923 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:04.709524 7923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:04.711122 7923 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:54:04.712069 7938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:04.712409 7923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:54:04.712522 7923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "8b8f701658e040b8a114896b4175b52a"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "348ac58b300a568c8d8e901d424b2711"
server_key_iv: "755095b338d098f0ee21e4ee584f1f27"
server_key_version: "encryptionkey@0"
I20260926 05:54:04.712960 7923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:04.728425 7923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:04.730818 7923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:04.730970 7923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:04.734889 7923 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:35393
I20260926 05:54:04.735074 7990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:35393 every 8 connection(s)
I20260926 05:54:04.735311 7923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:54:04.736748 7991 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:54:04.737416 7991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Bootstrap starting.
I20260926 05:54:04.743860 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 7923
I20260926 05:54:04.744174 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:38011
--local_ip_for_outbound_sockets=127.24.255.1
--tserver_master_addrs=127.24.255.62:35393
--webserver_port=35959
--webserver_interface=127.24.255.1
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:54:04.746973 7991 log.cc:824] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:04.750155 7991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:54:04.750658 7991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a: Bootstrap complete.
I20260926 05:54:04.754055 7991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } }
I20260926 05:54:04.754383 7991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8f701658e040b8a114896b4175b52a, State: Initialized, Role: FOLLOWER
I20260926 05:54:04.754565 7991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } }
I20260926 05:54:04.754643 7991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:54:04.754693 7991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:54:04.754740 7991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 2 FOLLOWER]: Advancing to term 3
I20260926 05:54:04.755793 7991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } }
I20260926 05:54:04.755891 7991 leader_election.cc:302] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b8f701658e040b8a114896b4175b52a; no voters:
I20260926 05:54:04.756059 7991 leader_election.cc:288] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 05:54:04.756110 7995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 3 FOLLOWER]: Leader election won for term 3
I20260926 05:54:04.756251 7995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [term 3 LEADER]: Becoming Leader. State: Replica: 8b8f701658e040b8a114896b4175b52a, State: Running, Role: LEADER
I20260926 05:54:04.756335 7991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:54:04.756421 7995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } }
I20260926 05:54:04.756785 7996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8b8f701658e040b8a114896b4175b52a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } } }
I20260926 05:54:04.756960 7996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: This master's current role is: LEADER
I20260926 05:54:04.756908 7995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b8f701658e040b8a114896b4175b52a. Latest consensus state: current_term: 3 leader_uuid: "8b8f701658e040b8a114896b4175b52a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8f701658e040b8a114896b4175b52a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35393 } } }
I20260926 05:54:04.757066 7995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8f701658e040b8a114896b4175b52a [sys.catalog]: This master's current role is: LEADER
I20260926 05:54:04.757366 8004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:54:04.758415 8004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:54:04.758519 8004 catalog_manager.cc:1259] Loaded cluster ID: 610c8d3c949e4198bdab12aa34776db1
I20260926 05:54:04.758560 8004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 05:54:04.758767 8004 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd8bd0fa9ad google::LogMessage::Fail() at ??:0
@ 0x7fd8bd0fd88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd8bd0fa4de google::LogMessage::Flush() at ??:0
@ 0x7fd8bd0fba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd8c501b0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd8c50248b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd8befc80d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd8befbd954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd8bfe216db start_thread at ??:0
@ 0x7fd8bbbf861f clone at ??:0
W20260926 05:54:04.856045 7993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:04.856268 7993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:04.856293 7993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:04.857527 7993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:04.857578 7993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:04.857607 7993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:54:04.859195 7993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:38011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=35959
--tserver_master_addrs=127.24.255.62:35393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:04.859450 7993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:04.859680 7993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:04.864567 8028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:04.864738 8027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:04.864861 8030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:04.865096 7993 server_base.cc:1034] running on GCE node
I20260926 05:54:04.865279 7993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:04.865520 7993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:04.866686 7993 hybrid_clock.cc:648] HybridClock initialized: now 1790402044866672 us; error 27 us; skew 500 ppm
Sep 26 05:54:04 dist-test-slave-7fr8 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402044, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:04.869575 7993 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:54:04.870158 7993 webserver.cc:460] Webserver started at http://127.24.255.1:35959/ using document root <none> and password file <none>
I20260926 05:54:04.870414 7993 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:04.870465 7993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:04.872457 7993 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:54:04.873103 8037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:04.873308 7993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:54:04.873413 7993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "fe6f885c67b14c259ba22f0b5d70c0bb"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "ee91f9207c24bbdedb3d2097e3a29985"
server_key_iv: "826f13fe95715f8b636acfeaecb502c2"
server_key_version: "encryptionkey@0"
I20260926 05:54:04.873876 7993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:04.900643 7993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:04.903362 7993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:04.903528 7993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:04.903795 7993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:04.904402 8044 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 05:54:04.905933 7993 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 05:54:04.906003 7993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 05:54:04.906039 7993 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 05:54:04.906903 7993 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 05:54:04.906947 7993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 05:54:04.907047 8044 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb: Bootstrap starting.
I20260926 05:54:04.914335 7993 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:38011
I20260926 05:54:04.914405 8151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:38011 every 8 connection(s)
I20260926 05:54:04.914795 7993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:54:04.917610 8044 log.cc:824] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:04.920251 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 7993
I20260926 05:54:04.920531 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:44457
--local_ip_for_outbound_sockets=127.24.255.2
--tserver_master_addrs=127.24.255.62:35393
--webserver_port=33267
--webserver_interface=127.24.255.2
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:54:04.920572 8044 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:54:04.920861 8044 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb: Bootstrap complete.
I20260926 05:54:04.921926 8044 ts_tablet_manager.cc:1397] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260926 05:54:04.923514 8044 raft_consensus.cc:348] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:04.923889 8044 raft_consensus.cc:729] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6f885c67b14c259ba22f0b5d70c0bb, State: Initialized, Role: FOLLOWER
I20260926 05:54:04.924063 8044 consensus_queue.cc:260] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:04.924397 8044 ts_tablet_manager.cc:1428] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb: Time spent starting tablet: real 0.002s user 0.004s sys 0.002s
W20260926 05:54:04.995951 8154 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:54:04.915520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:47359)
0926 05:54:04.915688 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:04.915757 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:04.916224 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:04.995738 (+ 79514us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
W20260926 05:54:04.996271 8152 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:35393 (0 consecutive failures): Network error: Failed to ping master at 127.24.255.62:35393: Client connection negotiation failed: client connection to 127.24.255.62:35393: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 05:54:04.996937 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.996652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:43521)
0926 05:54:04.996850 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:04.996906 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260926 05:54:04.997320 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:04.997228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:45099)
0926 05:54:04.997272 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:04.997303 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 05:54:04.997434 8152 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:54:05.013238 8156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:05.013470 8156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:05.013497 8156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:05.015480 8156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:05.015578 8156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:05.015619 8156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:54:05.018400 8156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:44457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=33267
--tserver_master_addrs=127.24.255.62:35393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:05.018682 8156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:05.018879 8156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:05.024395 8162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:05.024595 8156 server_base.cc:1034] running on GCE node
W20260926 05:54:05.024395 8165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:05.024395 8163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:05.024976 8156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:05.025242 8156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:05.026410 8156 hybrid_clock.cc:648] HybridClock initialized: now 1790402045026391 us; error 55 us; skew 500 ppm
Sep 26 05:54:05 dist-test-slave-7fr8 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402045, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:05.029584 8156 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:54:05.030153 8156 webserver.cc:460] Webserver started at http://127.24.255.2:33267/ using document root <none> and password file <none>
I20260926 05:54:05.030383 8156 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:05.030433 8156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:05.032106 8156 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:54:05.032915 8172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:05.033165 8156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:05.033260 8156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "b06214c9867b494c84313036fed37dcf"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "b7054c158ad0f29ee5fa5bc964ae3aae"
server_key_iv: "ddc912c8a8aab03b865f35e74a21cd2f"
server_key_version: "encryptionkey@0"
I20260926 05:54:05.033634 8156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:05.046519 8156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:05.049067 8156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:05.049222 8156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:05.049456 8156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:05.049901 8179 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 05:54:05.051565 8156 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 05:54:05.051687 8156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 05:54:05.051738 8156 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 05:54:05.052773 8156 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 05:54:05.052912 8156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 05:54:05.053076 8179 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: Bootstrap starting.
I20260926 05:54:05.060585 8156 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:44457
I20260926 05:54:05.060743 8286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:44457 every 8 connection(s)
I20260926 05:54:05.061080 8156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:54:05.062305 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:05.061765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:38349)
0926 05:54:05.061940 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:05.062041 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
W20260926 05:54:05.062649 8287 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:35393 (0 consecutive failures): Network error: Failed to ping master at 127.24.255.62:35393: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
I20260926 05:54:05.062930 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:05.062829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:60329)
0926 05:54:05.062873 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:05.062907 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 05:54:05.063103 8179 log.cc:824] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:05.063663 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:05.063458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:60299)
0926 05:54:05.063547 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:05.063641 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 05:54:05.063812 8287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:54:05.065260 8179 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:54:05.065517 8179 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: Bootstrap complete.
I20260926 05:54:05.066746 8179 ts_tablet_manager.cc:1397] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20260926 05:54:05.067891 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 8156
I20260926 05:54:05.068207 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:45331
--local_ip_for_outbound_sockets=127.24.255.3
--tserver_master_addrs=127.24.255.62:35393
--webserver_port=36499
--webserver_interface=127.24.255.3
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:54:05.069275 8179 raft_consensus.cc:348] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:05.069639 8179 raft_consensus.cc:729] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06214c9867b494c84313036fed37dcf, State: Initialized, Role: FOLLOWER
I20260926 05:54:05.069823 8179 consensus_queue.cc:260] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:05.070143 8179 ts_tablet_manager.cc:1428] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260926 05:54:05.070524 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:05.070405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:54863)
0926 05:54:05.070461 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:05.070500 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:54:05.070856 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:05.070775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:55743)
0926 05:54:05.070812 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:05.070835 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 05:54:05.155158 8291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:05.155400 8291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:05.155424 8291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:05.156682 8291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:05.156741 8291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:05.156770 8291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:54:05.158404 8291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:45331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=36499
--tserver_master_addrs=127.24.255.62:35393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:05.158751 8291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:05.158983 8291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:05.164266 8300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:05.164274 8297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:05.164383 8298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:05.164558 8291 server_base.cc:1034] running on GCE node
I20260926 05:54:05.164768 8291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:05.165017 8291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:05.166321 8291 hybrid_clock.cc:648] HybridClock initialized: now 1790402045166291 us; error 31 us; skew 500 ppm
Sep 26 05:54:05 dist-test-slave-7fr8 krb5kdc[6851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402045, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:05.169533 8291 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:54:05.170346 8291 webserver.cc:460] Webserver started at http://127.24.255.3:36499/ using document root <none> and password file <none>
I20260926 05:54:05.170781 8291 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:05.170895 8291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:05.172791 8291 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:54:05.173679 8307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:05.174069 8291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:05.174348 8291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30"
format_stamp: "Formatted at 2026-09-26 05:54:03 on dist-test-slave-7fr8"
server_key: "6dc9e846bba4ad644cafd68543b48882"
server_key_iv: "88b8aeb119e517c27e70cbc668582dbf"
server_key_version: "encryptionkey@0"
I20260926 05:54:05.174832 8291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:05.189941 8291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:05.192559 8291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:05.192709 8291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:05.192951 8291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:05.193264 8291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:54:05.193300 8291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:05.193331 8291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:54:05.193347 8291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:05.199481 8291 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:45331
I20260926 05:54:05.199537 8420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:45331 every 8 connection(s)
I20260926 05:54:05.199883 8291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:54:05.201192 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:05.200560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:42033)
0926 05:54:05.200817 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:05.200975 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":70,"threads_started":1}
W20260926 05:54:05.201808 8421 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:35393 (0 consecutive failures): Network error: Failed to ping master at 127.24.255.62:35393: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
I20260926 05:54:05.202270 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:05.202139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:45529)
0926 05:54:05.202206 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:05.202239 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 05:54:05.202888 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:05.202670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:59275)
0926 05:54:05.202816 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:05.202851 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260926 05:54:05.203202 8421 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:54:05.204887 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 8291
I20260926 05:54:05.998463 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:05.997946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:57719)
0926 05:54:05.998263 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:05.998337 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":72,"threads_started":1}
I20260926 05:54:06.071621 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:06.071303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:44135)
0926 05:54:06.071483 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:06.071557 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":68,"threads_started":1}
I20260926 05:54:06.203918 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:06.203649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:35341)
0926 05:54:06.203800 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:06.203846 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260926 05:54:06.404296 8428 raft_consensus.cc:482] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:54:06.404439 8428 raft_consensus.cc:504] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:06.404721 8428 leader_election.cc:288] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb45481d05e4dcfa6cc77e4d5bf5f30 (127.24.255.3:45331), fe6f885c67b14c259ba22f0b5d70c0bb (127.24.255.1:38011)
Sep 26 05:54:06 dist-test-slave-7fr8 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402045, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.1@KRBTEST.COM
Sep 26 05:54:06 dist-test-slave-7fr8 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402045, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.3@KRBTEST.COM
I20260926 05:54:06.416836 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:06.405340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:33331 (local address 127.24.255.1:38011)
0926 05:54:06.405599 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:06.405604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:06.405616 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:54:06.405861 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:06.405867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:06.405904 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:06.406042 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:06.406932 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:06.408371 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:06.409518 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:06.409982 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:06.410075 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:06.412987 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:06.413000 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:06.413000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:06.413020 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:06.415147 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:06.415604 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:06.415607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:06.415607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:06.415707 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:06.415973 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:06.415974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:06.415974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:06.416235 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:06.416309 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:06.416619 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 05:54:06.416741 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":132,"threads_started":1}
I20260926 05:54:06.416846 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:06.405236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:54105 (local address 127.24.255.3:45331)
0926 05:54:06.405379 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:06.405396 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:06.405893 (+ 497us) server_negotiation.cc:407] Connection header received
0926 05:54:06.406747 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:06.406752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:06.406803 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:06.406962 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:06.407277 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:06.408525 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:06.409497 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:06.409983 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:06.410082 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:06.413371 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:06.413375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:06.413375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:06.413386 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:06.415146 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:06.415774 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:06.415777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:06.415777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:06.415864 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:06.416038 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:06.416039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:06.416039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:06.416265 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:06.416334 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:06.416596 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 05:54:06.416731 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260926 05:54:06.416970 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:06.405072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:38011 (local address 127.24.255.2:33331)
0926 05:54:06.405144 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:06.405192 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:06.405583 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:06.406195 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:06.406199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:06.406531 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:06.406752 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:06.406758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:06.408526 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:06.408527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:06.409410 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:06.409413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:06.410075 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:06.410076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:06.410237 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:06.410878 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:06.410881 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:06.412822 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:06.415253 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:06.415263 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:06.415266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:06.415503 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:06.415800 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:06.415801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:06.415801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:06.415910 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:06.416301 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:06.416307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:06.416466 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 05:54:06.416533 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 05:54:06.416749 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":31}
I20260926 05:54:06.417268 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:06.405085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:45331 (local address 127.24.255.2:54105)
0926 05:54:06.405768 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:06.405781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:06.405831 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:06.406978 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:06.406978 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:06.407125 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:06.407177 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:06.407179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:06.408613 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:06.408615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:06.409414 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:06.409417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:06.410201 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:06.410202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:06.410249 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:06.410845 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:06.410856 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:06.413260 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:06.415572 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:06.415572 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:06.415573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:06.415696 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:06.415927 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:06.415927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:06.415928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:06.415984 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:06.416332 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:06.416333 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:06.416435 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 05:54:06.416533 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 05:54:06.416826 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":744,"thread_start_us":78,"threads_started":1}
I20260926 05:54:06.417414 8106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4184972c588b4621b1c2d4d35ccf2511" candidate_uuid: "b06214c9867b494c84313036fed37dcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" is_pre_election: true
I20260926 05:54:06.417577 8106 raft_consensus.cc:2454] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b06214c9867b494c84313036fed37dcf in term 1.
I20260926 05:54:06.417845 8176 leader_election.cc:302] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b06214c9867b494c84313036fed37dcf, fe6f885c67b14c259ba22f0b5d70c0bb; no voters:
I20260926 05:54:06.417963 8428 raft_consensus.cc:2781] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260926 05:54:06.417990 8173 leader_election.cc:341] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer feb45481d05e4dcfa6cc77e4d5bf5f30 (127.24.255.3:45331): Not found: Tablet not found: 4184972c588b4621b1c2d4d35ccf2511
I20260926 05:54:06.418015 8428 raft_consensus.cc:482] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:54:06.418074 8428 raft_consensus.cc:3037] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:54:06.419212 8428 raft_consensus.cc:504] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:06.419606 8428 leader_election.cc:288] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [CANDIDATE]: Term 2 election: Requested vote from peers feb45481d05e4dcfa6cc77e4d5bf5f30 (127.24.255.3:45331), fe6f885c67b14c259ba22f0b5d70c0bb (127.24.255.1:38011)
I20260926 05:54:06.419988 8106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4184972c588b4621b1c2d4d35ccf2511" candidate_uuid: "b06214c9867b494c84313036fed37dcf" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb"
W20260926 05:54:06.420153 8173 leader_election.cc:341] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer feb45481d05e4dcfa6cc77e4d5bf5f30 (127.24.255.3:45331): Not found: Tablet not found: 4184972c588b4621b1c2d4d35ccf2511
I20260926 05:54:06.420153 8106 raft_consensus.cc:3037] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:54:06.422067 8106 raft_consensus.cc:2454] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b06214c9867b494c84313036fed37dcf in term 2.
I20260926 05:54:06.422556 8176 leader_election.cc:302] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b06214c9867b494c84313036fed37dcf, fe6f885c67b14c259ba22f0b5d70c0bb; no voters: feb45481d05e4dcfa6cc77e4d5bf5f30
I20260926 05:54:06.422686 8428 raft_consensus.cc:2781] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:54:06.422834 8428 raft_consensus.cc:686] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [term 2 LEADER]: Becoming Leader. State: Replica: b06214c9867b494c84313036fed37dcf, State: Running, Role: LEADER
I20260926 05:54:06.422942 8428 consensus_queue.cc:237] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:06.423247 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:06.423120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:33721)
0926 05:54:06.423172 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:06.423211 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 05:54:06.892045 8106 raft_consensus.cc:1261] T 4184972c588b4621b1c2d4d35ccf2511 P fe6f885c67b14c259ba22f0b5d70c0bb [term 2 FOLLOWER]: Refusing update from remote peer b06214c9867b494c84313036fed37dcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260926 05:54:06.892503 8428 consensus_queue.cc:1035] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260926 05:54:06.893741 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:06.893256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:42971)
0926 05:54:06.893483 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:06.893635 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":43,"threads_started":1}
I20260926 05:54:06.894342 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:06.894238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:36593)
0926 05:54:06.894278 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:06.894306 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 05:54:06.894515 8173 consensus_peers.cc:487] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf -> Peer feb45481d05e4dcfa6cc77e4d5bf5f30 (127.24.255.3:45331): Couldn't send request to peer feb45481d05e4dcfa6cc77e4d5bf5f30. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4184972c588b4621b1c2d4d35ccf2511. This is attempt 1: this message will repeat every 5th retry.
I20260926 05:54:06.894622 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:06.894485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:47031)
0926 05:54:06.894552 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:06.894585 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 05:54:07.204784 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:07.204531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:55575)
0926 05:54:07.204689 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:07.204736 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260926 05:54:07.408862 8440 ts_tablet_manager.cc:927] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Initiating tablet copy from peer b06214c9867b494c84313036fed37dcf (127.24.255.2:44457)
I20260926 05:54:07.409252 8440 tablet_copy_client.cc:316] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: tablet copy: Beginning tablet copy session from remote peer at address 127.24.255.2:44457
Sep 26 05:54:07 dist-test-slave-7fr8 krb5kdc[6851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402045, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.2@KRBTEST.COM
I20260926 05:54:07.418462 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:07.409772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:44457 (local address 127.24.255.3:38095)
0926 05:54:07.409890 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:07.409910 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:07.410017 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:07.410210 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:07.410216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:07.410494 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:07.410583 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:07.410585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:07.411786 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:07.411787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:07.412497 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:07.412500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:07.413064 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:07.413065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:07.413135 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:07.413652 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:07.413656 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:07.415033 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:07.416805 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:07.416813 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:07.416816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:07.417039 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:07.417420 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:07.417423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:07.417423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:07.417559 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:07.418029 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:07.418035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:07.418140 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 05:54:07.418236 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 05:54:07.418356 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 05:54:07.418479 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:07.409848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:38095 (local address 127.24.255.2:44457)
0926 05:54:07.410010 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:07.410015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:07.410029 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:54:07.410074 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:07.410076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:07.410090 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:07.410194 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:07.410695 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:07.411688 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:07.412626 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:07.412950 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:07.413015 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:07.415140 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:07.415148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:07.415148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:07.415156 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:07.416708 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:07.417137 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:07.417140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:07.417140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:07.417208 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:07.417702 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:07.417703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:07.417704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:07.417917 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:07.418001 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:07.418304 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 05:54:07.418388 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260926 05:54:07.418898 8261 tablet_copy_service.cc:140] P b06214c9867b494c84313036fed37dcf: Received BeginTabletCopySession request for tablet 4184972c588b4621b1c2d4d35ccf2511 from peer feb45481d05e4dcfa6cc77e4d5bf5f30 ({username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:38095)
I20260926 05:54:07.418964 8261 tablet_copy_service.cc:161] P b06214c9867b494c84313036fed37dcf: Beginning new tablet copy session on tablet 4184972c588b4621b1c2d4d35ccf2511 from peer feb45481d05e4dcfa6cc77e4d5bf5f30 at {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:38095: session id = feb45481d05e4dcfa6cc77e4d5bf5f30-4184972c588b4621b1c2d4d35ccf2511
I20260926 05:54:07.419409 8261 tablet_copy_source_session.cc:205] T 4184972c588b4621b1c2d4d35ccf2511 P b06214c9867b494c84313036fed37dcf: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 05:54:07.420432 8440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4184972c588b4621b1c2d4d35ccf2511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:07.423277 8440 tablet_copy_client.cc:791] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: tablet copy: Starting download of 0 data blocks...
I20260926 05:54:07.423432 8440 tablet_copy_client.cc:655] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: tablet copy: Starting download of 1 WAL segments...
I20260926 05:54:07.424587 8440 tablet_copy_client.cc:531] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 05:54:07.425384 8440 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Bootstrap starting.
I20260926 05:54:07.426808 8440 log.cc:824] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:07.429430 8440 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:54:07.429656 8440 tablet_bootstrap.cc:492] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Bootstrap complete.
I20260926 05:54:07.429797 8440 ts_tablet_manager.cc:1397] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 05:54:07.430998 8440 raft_consensus.cc:348] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:07.431162 8440 raft_consensus.cc:729] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb45481d05e4dcfa6cc77e4d5bf5f30, State: Initialized, Role: FOLLOWER
I20260926 05:54:07.431269 8440 consensus_queue.cc:260] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb45481d05e4dcfa6cc77e4d5bf5f30" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 45331 } } peers { permanent_uuid: "b06214c9867b494c84313036fed37dcf" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 44457 } } peers { permanent_uuid: "fe6f885c67b14c259ba22f0b5d70c0bb" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38011 } }
I20260926 05:54:07.431541 8440 ts_tablet_manager.cc:1428] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260926 05:54:07.432157 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:07.432015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:47521)
0926 05:54:07.432068 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:07.432128 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 05:54:07.432253 8261 tablet_copy_service.cc:342] P b06214c9867b494c84313036fed37dcf: Request end of tablet copy session feb45481d05e4dcfa6cc77e4d5bf5f30-4184972c588b4621b1c2d4d35ccf2511 received from {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:38095
I20260926 05:54:07.432297 8261 tablet_copy_service.cc:433] P b06214c9867b494c84313036fed37dcf: ending tablet copy session feb45481d05e4dcfa6cc77e4d5bf5f30-4184972c588b4621b1c2d4d35ccf2511 on tablet 4184972c588b4621b1c2d4d35ccf2511 with peer feb45481d05e4dcfa6cc77e4d5bf5f30
I20260926 05:54:07.895241 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:07.894853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:55971)
0926 05:54:07.895132 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:07.895180 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260926 05:54:07.895442 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:07.895208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:36661)
0926 05:54:07.895348 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:07.895398 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":56,"threads_started":1}
I20260926 05:54:07.972301 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:07.972133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:59709)
0926 05:54:07.972216 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:07.972260 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 05:54:07.972292 8375 raft_consensus.cc:1203] T 4184972c588b4621b1c2d4d35ccf2511 P feb45481d05e4dcfa6cc77e4d5bf5f30 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260926 05:54:07.972864 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:07.972698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:39367)
0926 05:54:07.972791 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:07.972815 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":43,"threads_started":1}
I20260926 05:54:08.896272 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:08.896024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:33465)
0926 05:54:08.896175 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:08.896223 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260926 05:54:08.973096 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:08.972844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:35313)
0926 05:54:08.972987 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:08.973027 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260926 05:54:08.973438 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:08.973277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:40769)
0926 05:54:08.973371 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:08.973401 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":55,"threads_started":1}
I20260926 05:54:09.897092 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:09.896823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:40479)
0926 05:54:09.897004 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:09.897048 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":62,"threads_started":1}
I20260926 05:54:09.974037 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:09.973744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:40579)
0926 05:54:09.973924 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:09.973976 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260926 05:54:09.974197 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:09.973926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:39433)
0926 05:54:09.974065 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:09.974134 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260926 05:54:10.897993 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:10.897745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:60619)
0926 05:54:10.897892 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:10.897943 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260926 05:54:10.974840 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:10.974598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:47843)
0926 05:54:10.974767 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:10.974799 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260926 05:54:10.974840 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:10.974627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:50055)
0926 05:54:10.974749 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:10.974794 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260926 05:54:10.975198 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:10.975074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:33729)
0926 05:54:10.975140 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:10.975174 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 05:54:11.898861 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:11.898612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:55267)
0926 05:54:11.898768 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:11.898815 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260926 05:54:11.975636 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:11.975422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:46073)
0926 05:54:11.975557 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:11.975590 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260926 05:54:11.975795 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:11.975616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:59879)
0926 05:54:11.975709 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:11.975746 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":42,"threads_started":1}
I20260926 05:54:12.899857 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:12.899530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:38407)
0926 05:54:12.899745 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:12.899805 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":96,"threads_started":1}
I20260926 05:54:12.976413 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:12.976158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:47233)
0926 05:54:12.976307 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:12.976354 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260926 05:54:12.976471 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:12.976268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:41421)
0926 05:54:12.976387 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:12.976424 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260926 05:54:13.900807 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:13.900535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:58823)
0926 05:54:13.900706 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:13.900754 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":71,"threads_started":1}
I20260926 05:54:13.977579 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:13.977266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:35945)
0926 05:54:13.977427 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:13.977528 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":59,"threads_started":1}
I20260926 05:54:13.977552 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:13.977127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:36889)
0926 05:54:13.977415 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:13.977481 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":120,"threads_started":1}
I20260926 05:54:14.901731 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:14.901445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:60055)
0926 05:54:14.901638 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:14.901683 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
I20260926 05:54:14.978459 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:14.978210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:39039)
0926 05:54:14.978345 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:14.978389 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260926 05:54:14.978449 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:14.978202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:35505)
0926 05:54:14.978340 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:14.978390 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
I20260926 05:54:15.902657 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:15.902398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:56283)
0926 05:54:15.902552 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:15.902605 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260926 05:54:15.979223 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:15.978959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:45367)
0926 05:54:15.979140 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:15.979179 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":97,"threads_started":1}
I20260926 05:54:15.979223 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:15.978959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:49307)
0926 05:54:15.979128 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:15.979171 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":61,"threads_started":1}
I20260926 05:54:16.903484 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:16.903217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:32821)
0926 05:54:16.903392 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:16.903435 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
I20260926 05:54:16.980012 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:16.979788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:49955)
0926 05:54:16.979919 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:16.979952 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260926 05:54:16.980039 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:16.979788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:51511)
0926 05:54:16.979940 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:16.979974 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260926 05:54:17.904464 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:17.904148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:51845)
0926 05:54:17.904368 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:17.904412 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":135,"threads_started":1}
I20260926 05:54:17.980711 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:17.980481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:56155)
0926 05:54:17.980630 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:17.980667 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260926 05:54:17.980721 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:17.980479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:37737)
0926 05:54:17.980650 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:17.980681 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":116,"threads_started":1}
I20260926 05:54:18.905355 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:18.905070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:33785)
0926 05:54:18.905240 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:18.905288 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
I20260926 05:54:18.981570 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:18.981300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:57651)
0926 05:54:18.981485 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:18.981531 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
I20260926 05:54:18.981561 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:18.981321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:37223)
0926 05:54:18.981475 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:18.981512 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
I20260926 05:54:19.906328 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:19.906041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.1:56235)
0926 05:54:19.906227 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:19.906282 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
I20260926 05:54:19.982393 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:19.982169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.3:33493)
0926 05:54:19.982310 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:19.982348 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260926 05:54:19.982453 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:19.982258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35393 (local address 127.24.255.2:54479)
0926 05:54:19.982374 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:19.982408 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260926 05:54:20.208310 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 7993
I20260926 05:54:20.212621 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 8156
I20260926 05:54:20.216966 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 8291
2026-09-26T05:54:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18255 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:54:20 dist-test-slave-7fr8 krb5kdc[8491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:54:20 dist-test-slave-7fr8 krb5kdc[8491](info): set up 2 sockets
Sep 26 05:54:20 dist-test-slave-7fr8 krb5kdc[8491](info): commencing operation
krb5kdc: starting...
W20260926 05:54:21.190285 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:54:21 dist-test-slave-7fr8 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:54:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:54:21.321128 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:44677
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:44677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:54:21.406390 8504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:21.406635 8504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:21.406672 8504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:21.408036 8504 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 05:54:21.408113 8504 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 05:54:21.408133 8504 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 05:54:21.408149 8504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:54:21.408165 8504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:21.408185 8504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:21.408205 8504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:54:21.408224 8504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:54:21.409938 8504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:44677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:44677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:21.410257 8504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:21.410498 8504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:21.415759 8512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:21.415788 8510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:21.415759 8509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:21.415982 8504 server_base.cc:1034] running on GCE node
I20260926 05:54:21.416426 8504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:21.416761 8504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:21.417944 8504 hybrid_clock.cc:648] HybridClock initialized: now 1790402061417897 us; error 84 us; skew 500 ppm
Sep 26 05:54:21 dist-test-slave-7fr8 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:21.420930 8504 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:54:21.421465 8504 webserver.cc:460] Webserver started at http://127.24.255.62:36547/ using document root <none> and password file <none>
I20260926 05:54:21.421876 8504 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:21.422003 8504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:21.422228 8504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:54:21.423962 8504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "7588e6e0324946cd9fb04fa8c7446da9"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "5a9feea4f20bb2a423319392fd44f8fc"
server_key_iv: "9e8febdab757184dbad539d59657d172"
server_key_version: "encryptionkey@0"
I20260926 05:54:21.424445 8504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "7588e6e0324946cd9fb04fa8c7446da9"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "5a9feea4f20bb2a423319392fd44f8fc"
server_key_iv: "9e8febdab757184dbad539d59657d172"
server_key_version: "encryptionkey@0"
I20260926 05:54:21.426015 8504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 05:54:21.427431 8519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:21.427726 8504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:54:21.427830 8504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "7588e6e0324946cd9fb04fa8c7446da9"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "5a9feea4f20bb2a423319392fd44f8fc"
server_key_iv: "9e8febdab757184dbad539d59657d172"
server_key_version: "encryptionkey@0"
I20260926 05:54:21.427932 8504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:21.445868 8504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:21.448483 8504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:21.448647 8504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:21.453056 8504 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:44677
I20260926 05:54:21.453109 8571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:44677 every 8 connection(s)
I20260926 05:54:21.453452 8504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:54:21.454424 8572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:21.456606 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 8504
I20260926 05:54:21.456715 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:54:21.456899 25596 external_mini_cluster.cc:1428] Setting key 70b5c48ed821988e091bb9b8d76ed2d6
I20260926 05:54:21.456957 8572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Bootstrap starting.
I20260926 05:54:21.457634 8572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:21.457919 8572 log.cc:824] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:21.458587 8572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: No bootstrap required, opened a new log
I20260926 05:54:21.460412 8572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } }
I20260926 05:54:21.460565 8572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:21.460598 8572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7588e6e0324946cd9fb04fa8c7446da9, State: Initialized, Role: FOLLOWER
I20260926 05:54:21.460731 8572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } }
I20260926 05:54:21.460819 8572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:54:21.460858 8572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:54:21.460915 8572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:21.461709 8572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } }
I20260926 05:54:21.461818 8572 leader_election.cc:302] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7588e6e0324946cd9fb04fa8c7446da9; no voters:
I20260926 05:54:21.462018 8572 leader_election.cc:288] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:54:21.462168 8577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:54:21.462347 8572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:54:21.462378 8577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 1 LEADER]: Becoming Leader. State: Replica: 7588e6e0324946cd9fb04fa8c7446da9, State: Running, Role: LEADER
I20260926 05:54:21.462458 8577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } }
I20260926 05:54:21.462843 8577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7588e6e0324946cd9fb04fa8c7446da9. Latest consensus state: current_term: 1 leader_uuid: "7588e6e0324946cd9fb04fa8c7446da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } } }
I20260926 05:54:21.462997 8577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:54:21.463546 8586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 05:54:21 dist-test-slave-7fr8 krb5kdc[8491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:54:21.463804 8578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7588e6e0324946cd9fb04fa8c7446da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } } }
I20260926 05:54:21.463940 8578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:54:21.464048 8586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:54:21.466013 8586 catalog_manager.cc:1347] Generated new cluster ID: 1361df9a144345c3b63c2bca5bf48c21
I20260926 05:54:21.466073 8586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:54:21.468573 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:21.457457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.24.255.62:44677)
0926 05:54:21.457652 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:21.457660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:21.457686 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:54:21.458258 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:21.458266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:21.458318 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:21.458447 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:21.459122 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.460011 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.460780 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.461409 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.461539 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:21.464340 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:21.464345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:21.464345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:21.464360 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:21.466610 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:21.467037 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:21.467040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:21.467041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:21.467143 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:21.467310 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:21.467310 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:21.467311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:21.467623 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:21.467742 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:21.468196 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 05:54:21.468340 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260926 05:54:21.470467 8586 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:54:21.471027 8586 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:54:21.482493 8586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Generated new TSK 0
I20260926 05:54:21.482693 8586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:54:21.514297 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:44677
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:54:21.596920 8598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:21.597177 8598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:21.597218 8598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:21.598881 8598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:21.598943 8598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:21.598984 8598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:54:21.600644 8598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:44677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:21.600917 8598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:21.601153 8598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:21.606653 8606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:21.606688 8604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:21.606724 8603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:21.606885 8598 server_base.cc:1034] running on GCE node
I20260926 05:54:21.607105 8598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:21.607358 8598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:21.608541 8598 hybrid_clock.cc:648] HybridClock initialized: now 1790402061608524 us; error 25 us; skew 500 ppm
Sep 26 05:54:21 dist-test-slave-7fr8 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:21.611372 8598 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:54:21.611929 8598 webserver.cc:460] Webserver started at http://127.24.255.1:42819/ using document root <none> and password file <none>
I20260926 05:54:21.612169 8598 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:21.612221 8598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:21.612316 8598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:54:21.613376 8598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "72f33b442e874350a6f59c8bc7a2f9e5"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "125a7e54446c22ff7ad2259e2839390e"
server_key_iv: "a3854e50c16a9a34b6d3629af8220749"
server_key_version: "encryptionkey@0"
I20260926 05:54:21.613691 8598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "72f33b442e874350a6f59c8bc7a2f9e5"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "125a7e54446c22ff7ad2259e2839390e"
server_key_iv: "a3854e50c16a9a34b6d3629af8220749"
server_key_version: "encryptionkey@0"
I20260926 05:54:21.615149 8598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:54:21.616036 8613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:21.616300 8598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:21.616391 8598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "72f33b442e874350a6f59c8bc7a2f9e5"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "125a7e54446c22ff7ad2259e2839390e"
server_key_iv: "a3854e50c16a9a34b6d3629af8220749"
server_key_version: "encryptionkey@0"
I20260926 05:54:21.616456 8598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:21.630678 8598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:21.633996 8598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:21.634234 8598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:21.634533 8598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:21.635098 8598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:54:21.635170 8598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:21.635207 8598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:54:21.635224 8598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:21.640946 8598 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:40759
I20260926 05:54:21.641016 8726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:40759 every 8 connection(s)
I20260926 05:54:21.641325 8598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:54:21 dist-test-slave-7fr8 krb5kdc[8491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:54:21.650010 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 8598
I20260926 05:54:21.650141 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:54:21.650346 25596 external_mini_cluster.cc:1428] Setting key 3870547e6e4608d550f80fb402131324
I20260926 05:54:21.651351 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:21.642156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:53261 (local address 127.24.255.62:44677)
0926 05:54:21.642248 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:21.642250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:21.642397 (+ 147us) server_negotiation.cc:407] Connection header received
0926 05:54:21.642951 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:21.642953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:21.642967 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:21.643030 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:21.644074 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.644577 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.645614 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.645822 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.645877 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:21.648236 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:21.648238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:21.648238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:21.648246 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:21.649662 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:21.650166 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:21.650167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:21.650168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:21.650238 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:21.650526 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:21.650528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:21.650528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:21.650671 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:21.650739 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:21.651149 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 05:54:21.651244 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 05:54:21.651472 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:21.642041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:53261)
0926 05:54:21.642261 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:21.642310 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:21.642835 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:21.643124 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:21.643127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:21.643740 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:21.643958 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:21.643963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.644681 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.644681 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:21.645507 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:21.645510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.645901 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.645902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:21.646026 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:21.646606 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:21.646615 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:21.648111 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:21.649758 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:21.649768 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:21.649770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:21.650039 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:21.650308 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:21.650309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:21.650309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:21.650439 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:21.650776 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:21.650784 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:21.650961 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 05:54:21.651091 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 05:54:21.651276 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":117,"threads_started":1}
I20260926 05:54:21.651961 8727 heartbeater.cc:347] Connected to a master server at 127.24.255.62:44677
I20260926 05:54:21.652138 8727 heartbeater.cc:464] Registering TS with master...
I20260926 05:54:21.652344 8727 heartbeater.cc:511] Master 127.24.255.62:44677 requested a full tablet report, sending...
I20260926 05:54:21.652868 8536 ts_manager.cc:194] Registered new tserver with Master: 72f33b442e874350a6f59c8bc7a2f9e5 (127.24.255.1:40759)
I20260926 05:54:21.653465 8536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:53261
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:54:21.678494 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:44677
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:54:21.762725 8732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:21.762959 8732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:21.762985 8732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:21.764530 8732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:21.764600 8732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:21.764633 8732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:54:21.766373 8732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:44677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:21.766789 8732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:21.767066 8732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:21.772862 8737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:21.772857 8738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:21.773227 8740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:21.773275 8732 server_base.cc:1034] running on GCE node
I20260926 05:54:21.773550 8732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:21.773792 8732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:21.774968 8732 hybrid_clock.cc:648] HybridClock initialized: now 1790402061774960 us; error 36 us; skew 500 ppm
Sep 26 05:54:21 dist-test-slave-7fr8 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:21.777630 8732 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:54:21.778484 8732 webserver.cc:460] Webserver started at http://127.24.255.2:37187/ using document root <none> and password file <none>
I20260926 05:54:21.778931 8732 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:21.779021 8732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:21.779155 8732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:54:21.780268 8732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "f6d3d128fefa4f2297b187f60cb6f2a8"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "67a17c1e7fe95d8ce02b98dc4e99de85"
server_key_iv: "fb4fe315bae14c055e1b598eb448d8bf"
server_key_version: "encryptionkey@0"
I20260926 05:54:21.780610 8732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "f6d3d128fefa4f2297b187f60cb6f2a8"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "67a17c1e7fe95d8ce02b98dc4e99de85"
server_key_iv: "fb4fe315bae14c055e1b598eb448d8bf"
server_key_version: "encryptionkey@0"
I20260926 05:54:21.781910 8732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:54:21.782768 8747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:21.783001 8732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:54:21.783092 8732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "f6d3d128fefa4f2297b187f60cb6f2a8"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "67a17c1e7fe95d8ce02b98dc4e99de85"
server_key_iv: "fb4fe315bae14c055e1b598eb448d8bf"
server_key_version: "encryptionkey@0"
I20260926 05:54:21.783157 8732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:21.793823 8732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:21.795981 8732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:21.796129 8732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:21.796365 8732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:21.796720 8732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:54:21.796756 8732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:21.796785 8732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:54:21.796803 8732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:21.802778 8732 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:41731
I20260926 05:54:21.802877 8860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:41731 every 8 connection(s)
I20260926 05:54:21.803166 8732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:54:21.804061 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 8732
I20260926 05:54:21.804162 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:54:21.804349 25596 external_mini_cluster.cc:1428] Setting key 4d8b563455c377a6ca01b2f664b3f4af
Sep 26 05:54:21 dist-test-slave-7fr8 krb5kdc[8491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
I20260926 05:54:21.816470 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:21.803992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:34831 (local address 127.24.255.62:44677)
0926 05:54:21.804051 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:21.804053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:21.804275 (+ 222us) server_negotiation.cc:407] Connection header received
0926 05:54:21.804920 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:21.804921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:21.804933 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:21.804996 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:21.806065 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.806833 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.808655 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.808855 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.808946 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:21.811524 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:21.811527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:21.811527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:21.811536 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:21.813565 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:21.814325 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:21.814327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:21.814327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:21.814405 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:21.814929 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:21.814932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:21.814933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:21.815411 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:21.815541 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:21.816118 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 05:54:21.816336 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:54:21.816672 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:21.803957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:34831)
0926 05:54:21.804141 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:21.804192 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:21.804813 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:21.805153 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:21.805157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:21.805743 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:21.805944 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:21.805949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.807048 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.807050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:21.808527 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:21.808533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.808956 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.808957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:21.809126 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:21.809696 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:21.809707 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:21.811407 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:21.813697 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:21.813709 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:21.813713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:21.814241 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:21.814508 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:21.814509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:21.814510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:21.814703 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:21.815569 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:21.815587 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:21.815897 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 05:54:21.816056 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 05:54:21.816403 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260926 05:54:21.817435 8861 heartbeater.cc:347] Connected to a master server at 127.24.255.62:44677
I20260926 05:54:21.817569 8861 heartbeater.cc:464] Registering TS with master...
I20260926 05:54:21.817781 8861 heartbeater.cc:511] Master 127.24.255.62:44677 requested a full tablet report, sending...
I20260926 05:54:21.818307 8536 ts_manager.cc:194] Registered new tserver with Master: f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731)
I20260926 05:54:21.818905 8536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:34831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:54:21.833665 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:44677
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:54:21.915925 8866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:21.916157 8866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:21.916184 8866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:21.917369 8866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:21.917411 8866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:21.917439 8866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:54:21.918972 8866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:44677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:21.919180 8866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:21.919368 8866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:21.925411 8874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:21.925419 8871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:21.925668 8866 server_base.cc:1034] running on GCE node
W20260926 05:54:21.925526 8872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:21.925971 8866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:21.926277 8866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:21.927459 8866 hybrid_clock.cc:648] HybridClock initialized: now 1790402061927437 us; error 38 us; skew 500 ppm
Sep 26 05:54:21 dist-test-slave-7fr8 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:21.930553 8866 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:54:21.931095 8866 webserver.cc:460] Webserver started at http://127.24.255.3:35899/ using document root <none> and password file <none>
I20260926 05:54:21.931324 8866 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:21.931376 8866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:21.931521 8866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:54:21.932714 8866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "4a0ea34d1a2540cb969e9b49c28d6596"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "554243cad2f7353cbc516ba67ea646f7"
server_key_iv: "05c3641a35f0036092c10851fa508474"
server_key_version: "encryptionkey@0"
I20260926 05:54:21.933073 8866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "4a0ea34d1a2540cb969e9b49c28d6596"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "554243cad2f7353cbc516ba67ea646f7"
server_key_iv: "05c3641a35f0036092c10851fa508474"
server_key_version: "encryptionkey@0"
I20260926 05:54:21.934623 8866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:54:21.935895 8881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:21.936205 8866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:21.936317 8866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "4a0ea34d1a2540cb969e9b49c28d6596"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "554243cad2f7353cbc516ba67ea646f7"
server_key_iv: "05c3641a35f0036092c10851fa508474"
server_key_version: "encryptionkey@0"
I20260926 05:54:21.936429 8866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:21.952811 8866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:21.955329 8866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:21.955495 8866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:21.955729 8866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:21.956058 8866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:54:21.956092 8866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:21.956123 8866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:54:21.956141 8866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:21.963152 8866 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:44513
I20260926 05:54:21.963202 8994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:44513 every 8 connection(s)
I20260926 05:54:21.963558 8866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:54:21.969832 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 8866
I20260926 05:54:21.969913 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:54:21.970130 25596 external_mini_cluster.cc:1428] Setting key 7f6869e0f8dd1f16967b418c548c6cdd
Sep 26 05:54:21 dist-test-slave-7fr8 krb5kdc[8491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:54:21.973753 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:21.964382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:48153 (local address 127.24.255.62:44677)
0926 05:54:21.964460 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:21.964462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:21.964633 (+ 171us) server_negotiation.cc:407] Connection header received
0926 05:54:21.965175 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:21.965176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:21.965187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:21.965242 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:21.966271 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.966844 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.967990 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.968212 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.968278 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:21.971029 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:21.971031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:21.971031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:21.971039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:21.972356 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:21.972815 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:21.972817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:21.972817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:21.972871 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:21.973104 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:21.973105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:21.973106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:21.973229 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:21.973286 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:21.973568 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 05:54:21.973657 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:54:21.973940 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:21.964320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:48153)
0926 05:54:21.964486 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:21.964529 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:21.965035 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:21.965301 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:21.965304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:21.965902 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:21.966182 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:21.966189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.966948 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.966949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:21.967895 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:21.967898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.968338 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.968340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:21.968510 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:21.969126 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:21.969142 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:21.970935 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:21.972460 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:21.972471 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:21.972473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:21.972742 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:21.972927 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:21.972928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:21.972929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:21.973048 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:21.973287 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:21.973294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:21.973445 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 05:54:21.973539 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 05:54:21.973710 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260926 05:54:21.974483 8995 heartbeater.cc:347] Connected to a master server at 127.24.255.62:44677
I20260926 05:54:21.974570 8995 heartbeater.cc:464] Registering TS with master...
I20260926 05:54:21.974756 8995 heartbeater.cc:511] Master 127.24.255.62:44677 requested a full tablet report, sending...
I20260926 05:54:21.975258 8536 ts_manager.cc:194] Registered new tserver with Master: 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513)
I20260926 05:54:21.975701 8536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:48153
I20260926 05:54:21.982478 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:54:21.991703 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:21.984000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39964 (local address 127.24.255.62:44677)
0926 05:54:21.984090 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:21.984093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:21.984282 (+ 189us) server_negotiation.cc:407] Connection header received
0926 05:54:21.984313 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:21.984314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:21.984326 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:21.984389 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:21.985158 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.985733 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.986723 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:21.986952 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:21.987017 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:21.988401 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:21.988402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:21.988402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:21.988409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:21.989813 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:21.990541 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:21.990542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:21.990542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:21.990614 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:21.990928 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:21.990929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:21.990930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:21.991101 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:21.991185 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:21.991501 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 05:54:21.991598 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 05:54:21.994004 8536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:54:21.996618 8536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:54:22.004190 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:22.000103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.24.255.2:41731)
0926 05:54:22.000299 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:22.000304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:22.000346 (+ 42us) server_negotiation.cc:407] Connection header received
0926 05:54:22.000379 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:22.000381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:22.000394 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:22.000731 (+ 337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:22.000795 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.001830 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.003290 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.003780 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.003846 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:22.003908 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:22.004070 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 05:54:22.004115 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260926 05:54:22.004204 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:22.000132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:44513 (local address 127.0.0.1:33432)
0926 05:54:22.000454 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:22.000463 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:22.000508 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:22.000743 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:22.000744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:22.000745 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:22.000776 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:22.000777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.001650 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.001651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:22.002956 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:22.002959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.003944 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.003945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:22.004010 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:22.004016 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:54:22.004061 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 05:54:22.004110 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":4,"thread_start_us":70,"threads_started":1}
I20260926 05:54:22.004282 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:22.000097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49576 (local address 127.24.255.1:40759)
0926 05:54:22.000301 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:22.000304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:22.000319 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:54:22.000362 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:22.000364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:22.000379 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:22.000465 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:22.000667 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.001829 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.003004 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.003778 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.003849 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:22.003908 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:22.004159 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 05:54:22.004208 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260926 05:54:22.004515 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:22.000247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.24.255.3:44513)
0926 05:54:22.000405 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:22.000410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:22.000554 (+ 144us) server_negotiation.cc:407] Connection header received
0926 05:54:22.000594 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:22.000596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:22.000611 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:22.000676 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:22.000843 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.001551 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.003036 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.003778 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.003849 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:22.003908 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:22.004412 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 05:54:22.004458 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260926 05:54:22.004709 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:21.999971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:41731 (local address 127.0.0.1:42360)
0926 05:54:22.000246 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:22.000256 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:22.000302 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:22.000494 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:22.000495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:22.000497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:22.000592 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:22.000595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.001938 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.001939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:22.003212 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:22.003214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.003866 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.003867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:22.003968 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:22.003980 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 05:54:22.004043 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 05:54:22.004108 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":65,"thread_start_us":69,"threads_started":1}
I20260926 05:54:22.004819 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:21.999777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:40759 (local address 127.0.0.1:49576)
0926 05:54:21.999901 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:21.999950 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:22.000018 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:22.000485 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:22.000488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:22.000491 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:22.000599 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:22.000601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.001963 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.001964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:22.002930 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:22.002933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.003951 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.003952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:22.004010 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:22.004014 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:54:22.004070 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 05:54:22.004109 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260926 05:54:22.005271 8929 tablet_service.cc:1467] Processing CreateTablet for tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab (DEFAULT_TABLE table=test-table [id=256f5ea2913d46d69e6f0d3e406ba50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:54:22.005295 8795 tablet_service.cc:1467] Processing CreateTablet for tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab (DEFAULT_TABLE table=test-table [id=256f5ea2913d46d69e6f0d3e406ba50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:54:22.005657 8795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:22.005657 8929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:22.006728 8661 tablet_service.cc:1467] Processing CreateTablet for tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab (DEFAULT_TABLE table=test-table [id=256f5ea2913d46d69e6f0d3e406ba50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:54:22.007037 8661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:22.009298 9012 tablet_bootstrap.cc:492] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596: Bootstrap starting.
I20260926 05:54:22.009779 9014 tablet_bootstrap.cc:492] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Bootstrap starting.
I20260926 05:54:22.009949 9013 tablet_bootstrap.cc:492] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8: Bootstrap starting.
I20260926 05:54:22.010390 9014 tablet_bootstrap.cc:654] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:22.010502 9013 tablet_bootstrap.cc:654] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:22.010875 9014 log.cc:824] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:22.010958 9013 log.cc:824] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:22.011447 9012 tablet_bootstrap.cc:654] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:22.011905 9012 log.cc:824] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:22.011974 9014 tablet_bootstrap.cc:492] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: No bootstrap required, opened a new log
I20260926 05:54:22.012097 9014 ts_tablet_manager.cc:1397] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:54:22.012671 9012 tablet_bootstrap.cc:492] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596: No bootstrap required, opened a new log
I20260926 05:54:22.012741 9012 ts_tablet_manager.cc:1397] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 05:54:22.013968 9014 raft_consensus.cc:348] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } }
I20260926 05:54:22.014243 9014 raft_consensus.cc:374] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:22.014297 9014 raft_consensus.cc:729] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f33b442e874350a6f59c8bc7a2f9e5, State: Initialized, Role: FOLLOWER
I20260926 05:54:22.014283 9012 raft_consensus.cc:348] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } }
I20260926 05:54:22.014492 9012 raft_consensus.cc:374] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:22.014456 9014 consensus_queue.cc:260] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } }
I20260926 05:54:22.014537 9012 raft_consensus.cc:729] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a0ea34d1a2540cb969e9b49c28d6596, State: Initialized, Role: FOLLOWER
I20260926 05:54:22.014688 9012 consensus_queue.cc:260] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } }
I20260926 05:54:22.014981 9012 ts_tablet_manager.cc:1428] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:54:22.015013 8995 heartbeater.cc:503] Master 127.24.255.62:44677 was elected leader, sending a full tablet report...
I20260926 05:54:22.015007 9014 ts_tablet_manager.cc:1428] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 05:54:22.015156 8727 heartbeater.cc:503] Master 127.24.255.62:44677 was elected leader, sending a full tablet report...
I20260926 05:54:22.015713 9013 tablet_bootstrap.cc:492] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8: No bootstrap required, opened a new log
I20260926 05:54:22.015798 9013 ts_tablet_manager.cc:1397] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 05:54:22.017375 9013 raft_consensus.cc:348] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } }
I20260926 05:54:22.017541 9013 raft_consensus.cc:374] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:22.017572 9013 raft_consensus.cc:729] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3d128fefa4f2297b187f60cb6f2a8, State: Initialized, Role: FOLLOWER
I20260926 05:54:22.017685 9013 consensus_queue.cc:260] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } }
I20260926 05:54:22.017979 9013 ts_tablet_manager.cc:1428] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:54:22.018138 8861 heartbeater.cc:503] Master 127.24.255.62:44677 was elected leader, sending a full tablet report...
W20260926 05:54:22.053805 8862 tablet.cc:2369] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:54:22.070336 9018 raft_consensus.cc:482] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:54:22.070544 9018 raft_consensus.cc:504] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } }
I20260926 05:54:22.070945 9018 leader_election.cc:288] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731), 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513)
I20260926 05:54:22.073946 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:22.071523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:49513 (local address 127.24.255.3:44513)
0926 05:54:22.071576 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:22.071578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:22.071608 (+ 30us) server_negotiation.cc:407] Connection header received
0926 05:54:22.071667 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:22.071667 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:22.071676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:22.071724 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:22.071870 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.072329 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.073238 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.073624 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.073689 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:22.073698 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:22.073841 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 05:54:22.073878 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:54:22.073966 9021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:22.071367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:44513 (local address 127.24.255.1:49513)
0926 05:54:22.071551 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:22.071561 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:22.071614 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:22.071732 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:22.071733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:22.071734 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:22.071804 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:22.071806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.072408 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.072409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:22.073145 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:22.073146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.073702 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.073703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:22.073774 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:22.073777 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:54:22.073822 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 05:54:22.073865 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":123,"threads_started":1}
I20260926 05:54:22.074023 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:22.071332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:45681 (local address 127.24.255.2:41731)
0926 05:54:22.071429 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:22.071431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:22.071442 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:54:22.071474 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:22.071475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:22.071490 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:22.071561 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:22.071734 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.072208 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.073171 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.073662 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.073744 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:22.073753 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:22.073913 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 05:54:22.073960 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 05:54:22.074244 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:22.071232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:41731 (local address 127.24.255.1:45681)
0926 05:54:22.071292 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:22.071305 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:22.071365 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:22.071588 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:22.071589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:22.071591 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:22.071660 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:22.071662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.072297 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.072298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:22.073063 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:22.073065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.073767 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.073768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:22.073844 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:22.073847 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:54:22.073895 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:54:22.073939 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:54:22.074445 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e3ffcdbf1f49a2a5e72581b0bd50ab" candidate_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" is_pre_election: true
I20260926 05:54:22.074548 8815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e3ffcdbf1f49a2a5e72581b0bd50ab" candidate_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" is_pre_election: true
I20260926 05:54:22.074602 8949 raft_consensus.cc:2454] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72f33b442e874350a6f59c8bc7a2f9e5 in term 0.
I20260926 05:54:22.074698 8815 raft_consensus.cc:2454] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72f33b442e874350a6f59c8bc7a2f9e5 in term 0.
I20260926 05:54:22.074824 8616 leader_election.cc:302] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a0ea34d1a2540cb969e9b49c28d6596, 72f33b442e874350a6f59c8bc7a2f9e5; no voters:
I20260926 05:54:22.075122 9018 raft_consensus.cc:2781] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:54:22.075198 9018 raft_consensus.cc:482] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:54:22.075223 9018 raft_consensus.cc:3037] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:22.076007 9018 raft_consensus.cc:504] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } }
I20260926 05:54:22.076135 9018 leader_election.cc:288] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 1 election: Requested vote from peers f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731), 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513)
I20260926 05:54:22.076365 8815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e3ffcdbf1f49a2a5e72581b0bd50ab" candidate_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8"
I20260926 05:54:22.076387 8949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e3ffcdbf1f49a2a5e72581b0bd50ab" candidate_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a0ea34d1a2540cb969e9b49c28d6596"
I20260926 05:54:22.076465 8815 raft_consensus.cc:3037] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:22.076514 8949 raft_consensus.cc:3037] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:22.077322 8949 raft_consensus.cc:2454] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72f33b442e874350a6f59c8bc7a2f9e5 in term 1.
I20260926 05:54:22.077317 8815 raft_consensus.cc:2454] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72f33b442e874350a6f59c8bc7a2f9e5 in term 1.
I20260926 05:54:22.077533 8616 leader_election.cc:302] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72f33b442e874350a6f59c8bc7a2f9e5, f6d3d128fefa4f2297b187f60cb6f2a8; no voters:
I20260926 05:54:22.077618 9018 raft_consensus.cc:2781] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:54:22.077792 9018 raft_consensus.cc:686] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 LEADER]: Becoming Leader. State: Replica: 72f33b442e874350a6f59c8bc7a2f9e5, State: Running, Role: LEADER
I20260926 05:54:22.077912 9018 consensus_queue.cc:237] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } }
I20260926 05:54:22.078833 8536 catalog_manager.cc:5495] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72f33b442e874350a6f59c8bc7a2f9e5 (127.24.255.1). New cstate: current_term: 1 leader_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } health_report { overall_health: UNKNOWN } } }
I20260926 05:54:22.099063 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:22.096740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.24.255.1:40759)
0926 05:54:22.096819 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:22.096820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:22.096827 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:54:22.096863 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:22.096865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:22.096884 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:22.096942 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:54:22.097078 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.097561 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.098335 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:22.098526 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:22.098581 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:22.098728 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:54:22.098835 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:54:22.098908 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:22.098929 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 05:54:22.098950 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 05:54:22.101122 8949 raft_consensus.cc:1261] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596 [term 1 FOLLOWER]: Refusing update from remote peer 72f33b442e874350a6f59c8bc7a2f9e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:54:22.101122 8815 raft_consensus.cc:1261] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8 [term 1 FOLLOWER]: Refusing update from remote peer 72f33b442e874350a6f59c8bc7a2f9e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:54:22.101727 9022 consensus_queue.cc:1035] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:54:22.101876 9018 consensus_queue.cc:1035] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:54:22.104930 9026 mvcc.cc:205] Tried to move back new op lower bound from 7333486846362583040 to 7333486846271836160. Current Snapshot: MvccSnapshot[applied={T|T < 7333486846362583040}]
I20260926 05:54:22.107164 8535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:54:22.107270 8535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:54:22.108165 8535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Sending DeleteTablet for 3 replicas of tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab
I20260926 05:54:22.108569 8661 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:54:22 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:49576
I20260926 05:54:22.108588 8929 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:54:22 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:33432
I20260926 05:54:22.108600 8795 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:54:22 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:42360
I20260926 05:54:22.108853 9034 tablet_replica.cc:330] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: stopping tablet replica
I20260926 05:54:22.108861 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 8598
I20260926 05:54:22.108961 9034 raft_consensus.cc:2229] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:54:22.109090 9034 raft_consensus.cc:2258] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:54:22.109484 9036 tablet_replica.cc:330] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8: stopping tablet replica
I20260926 05:54:22.109588 9036 raft_consensus.cc:2229] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:54:22.109689 9036 raft_consensus.cc:2258] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:54:22.110234 9036 ts_tablet_manager.cc:1905] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:54:22.110311 9035 tablet_replica.cc:330] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596: stopping tablet replica
I20260926 05:54:22.110430 9035 raft_consensus.cc:2229] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:54:22.110535 9035 raft_consensus.cc:2258] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:54:22.111161 9035 ts_tablet_manager.cc:1905] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:54:22.112042 9036 ts_tablet_manager.cc:1918] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:54:22.112111 9036 log.cc:1196] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/a3e3ffcdbf1f49a2a5e72581b0bd50ab
I20260926 05:54:22.112317 9036 ts_tablet_manager.cc:1939] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P f6d3d128fefa4f2297b187f60cb6f2a8: Deleting consensus metadata
I20260926 05:54:22.112771 8522 catalog_manager.cc:4841] TS f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731): tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab (table test-table [id=256f5ea2913d46d69e6f0d3e406ba50c]) successfully deleted
I20260926 05:54:22.113168 9035 ts_tablet_manager.cc:1918] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:54:22.113256 9035 log.cc:1196] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/a3e3ffcdbf1f49a2a5e72581b0bd50ab
I20260926 05:54:22.113533 9035 ts_tablet_manager.cc:1939] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 4a0ea34d1a2540cb969e9b49c28d6596: Deleting consensus metadata
I20260926 05:54:22.114048 8522 catalog_manager.cc:4841] TS 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513): tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab (table test-table [id=256f5ea2913d46d69e6f0d3e406ba50c]) successfully deleted
W20260926 05:54:22.115231 8522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:40759 (error 108)
I20260926 05:54:22.115437 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 8732
I20260926 05:54:22.117863 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:22.117671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:40759 (local address 127.0.0.1:49596)
0926 05:54:22.117758 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:22.117817 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:40759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 05:54:22.118026 8522 catalog_manager.cc:4584] TS 72f33b442e874350a6f59c8bc7a2f9e5 (127.24.255.1:40759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab: Network error: Client connection negotiation failed: client connection to 127.24.255.1:40759: connect: Connection refused (error 111)
I20260926 05:54:22.120951 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 8866
I20260926 05:54:22.125723 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 8504
I20260926 05:54:42.130837 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:44677
--webserver_interface=127.24.255.62
--webserver_port=36547
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:44677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:54:42.218214 9038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:42.218453 9038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:42.218477 9038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:42.219743 9038 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 05:54:42.219797 9038 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 05:54:42.219810 9038 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 05:54:42.219820 9038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:54:42.219831 9038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:42.219842 9038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:42.219853 9038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:54:42.219867 9038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:54:42.221504 9038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:44677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:44677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:42.221777 9038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:42.222012 9038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:42.231365 9047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:42.231370 9045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:42.231422 9038 server_base.cc:1034] running on GCE node
W20260926 05:54:42.231369 9044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:42.231931 9038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:42.232270 9038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:42.233529 9038 hybrid_clock.cc:648] HybridClock initialized: now 1790402082233492 us; error 44 us; skew 500 ppm
Sep 26 05:54:42 dist-test-slave-7fr8 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:42.236714 9038 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:54:42.237210 9038 webserver.cc:460] Webserver started at http://127.24.255.62:36547/ using document root <none> and password file <none>
I20260926 05:54:42.237439 9038 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:42.237506 9038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:42.239107 9038 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:42.239866 9054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:42.240152 9038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:42.240257 9038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "7588e6e0324946cd9fb04fa8c7446da9"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "5a9feea4f20bb2a423319392fd44f8fc"
server_key_iv: "9e8febdab757184dbad539d59657d172"
server_key_version: "encryptionkey@0"
I20260926 05:54:42.241127 9038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:42.299837 9038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:42.302398 9038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:42.302553 9038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:42.306640 9038 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:44677
I20260926 05:54:42.306850 9106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:44677 every 8 connection(s)
I20260926 05:54:42.307153 9038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:54:42.308877 9107 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:54:42.309587 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Bootstrap starting.
I20260926 05:54:42.317406 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 9038
I20260926 05:54:42.317736 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:40759
--local_ip_for_outbound_sockets=127.24.255.1
--tserver_master_addrs=127.24.255.62:44677
--webserver_port=42819
--webserver_interface=127.24.255.1
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:54:42.317911 9107 log.cc:824] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:42.320467 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:54:42.320760 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Bootstrap complete.
I20260926 05:54:42.323077 9107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } }
I20260926 05:54:42.323336 9107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7588e6e0324946cd9fb04fa8c7446da9, State: Initialized, Role: FOLLOWER
I20260926 05:54:42.323511 9107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } }
I20260926 05:54:42.323589 9107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:54:42.323647 9107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:54:42.323690 9107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:54:42.324817 9107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } }
I20260926 05:54:42.324922 9107 leader_election.cc:302] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7588e6e0324946cd9fb04fa8c7446da9; no voters:
I20260926 05:54:42.325094 9107 leader_election.cc:288] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 05:54:42.325170 9111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:54:42.325354 9107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:54:42.325412 9111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 2 LEADER]: Becoming Leader. State: Replica: 7588e6e0324946cd9fb04fa8c7446da9, State: Running, Role: LEADER
I20260926 05:54:42.325549 9111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } }
I20260926 05:54:42.326578 9112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7588e6e0324946cd9fb04fa8c7446da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } } }
I20260926 05:54:42.327222 9126 catalog_manager.cc:1259] Loaded cluster ID: 1361df9a144345c3b63c2bca5bf48c21
I20260926 05:54:42.327289 9126 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: loading cluster ID for follower catalog manager: success
I20260926 05:54:42.326814 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7588e6e0324946cd9fb04fa8c7446da9. Latest consensus state: current_term: 2 leader_uuid: "7588e6e0324946cd9fb04fa8c7446da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } } }
I20260926 05:54:42.327488 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:54:42.327239 9112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:54:42.328750 9126 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: acquiring CA information for follower catalog manager: success
I20260926 05:54:42.329227 9126 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:54:42.330315 9129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:54:42.330917 9129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:54:42.331013 9129 catalog_manager.cc:1259] Loaded cluster ID: 1361df9a144345c3b63c2bca5bf48c21
I20260926 05:54:42.331071 9129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:54:42.331267 9129 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:54:42.331344 9129 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Loaded TSK: 0
I20260926 05:54:42.340207 9129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Generated new TSK 1
I20260926 05:54:42.340338 9129 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 05:54:42.414726 9110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:42.414983 9110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:42.415015 9110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:42.417438 9110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:42.417582 9110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:42.417623 9110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:54:42.419872 9110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:40759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=42819
--tserver_master_addrs=127.24.255.62:44677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:42.420171 9110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:42.420430 9110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:42.426297 9139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:42.426278 9136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:42.426270 9135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:42.426638 9110 server_base.cc:1034] running on GCE node
I20260926 05:54:42.426856 9110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:42.427310 9110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:42.428501 9110 hybrid_clock.cc:648] HybridClock initialized: now 1790402082428485 us; error 42 us; skew 500 ppm
Sep 26 05:54:42 dist-test-slave-7fr8 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:42.431867 9110 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:54:42.432374 9110 webserver.cc:460] Webserver started at http://127.24.255.1:42819/ using document root <none> and password file <none>
I20260926 05:54:42.432605 9110 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:42.432665 9110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:42.434180 9110 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:54:42.434959 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:42.435185 9110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:42.435280 9110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "72f33b442e874350a6f59c8bc7a2f9e5"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "125a7e54446c22ff7ad2259e2839390e"
server_key_iv: "a3854e50c16a9a34b6d3629af8220749"
server_key_version: "encryptionkey@0"
I20260926 05:54:42.435657 9110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:42.465526 9110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:42.467746 9110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:42.467891 9110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:42.468133 9110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:42.468714 9153 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 05:54:42.470234 9110 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 05:54:42.470294 9110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 05:54:42.470332 9110 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 05:54:42.470990 9110 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 05:54:42.471026 9110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 05:54:42.471117 9153 tablet_bootstrap.cc:492] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Bootstrap starting.
I20260926 05:54:42.477123 9110 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:40759
I20260926 05:54:42.477233 9260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:40759 every 8 connection(s)
I20260926 05:54:42.477550 9110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:54:42.481482 9153 log.cc:824] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:42.483655 9153 tablet_bootstrap.cc:492] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:54:42.483958 9153 tablet_bootstrap.cc:492] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Bootstrap complete.
I20260926 05:54:42.484649 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 9110
I20260926 05:54:42.484929 9153 ts_tablet_manager.cc:1397] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260926 05:54:42.484941 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:41731
--local_ip_for_outbound_sockets=127.24.255.2
--tserver_master_addrs=127.24.255.62:44677
--webserver_port=37187
--webserver_interface=127.24.255.2
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:54:42 dist-test-slave-7fr8 krb5kdc[8491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:54:42.486208 9153 raft_consensus.cc:348] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } }
I20260926 05:54:42.486541 9153 raft_consensus.cc:729] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f33b442e874350a6f59c8bc7a2f9e5, State: Initialized, Role: FOLLOWER
I20260926 05:54:42.486712 9153 consensus_queue.cc:260] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } }
I20260926 05:54:42.486992 9153 ts_tablet_manager.cc:1428] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260926 05:54:42.489302 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.478397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:38853 (local address 127.24.255.62:44677)
0926 05:54:42.478551 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:42.478559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:42.478587 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:54:42.479414 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:42.479420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:42.479523 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:42.479636 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:42.480653 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.481609 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.482678 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.483151 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.483248 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.485802 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:42.485807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:42.485807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:42.485822 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:42.487640 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:42.488041 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:42.488045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:42.488045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:42.488154 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:42.488382 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:42.488383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:42.488383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:42.488624 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:42.488698 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:42.488974 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 05:54:42.489113 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260926 05:54:42.489302 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.478236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:38853)
0926 05:54:42.478431 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:42.478488 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:42.479151 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:42.479737 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:42.479742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:42.480177 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:42.480501 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.480510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.481737 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.481738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.482597 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.482600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.483247 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.483249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.483383 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.483871 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:42.483890 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:42.485667 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:42.487729 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:42.487736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:42.487738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:42.487960 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:42.488221 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:42.488222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:42.488222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:42.488321 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:42.488690 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:42.488696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:42.488825 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 05:54:42.488914 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 05:54:42.489100 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260926 05:54:42.490196 9261 heartbeater.cc:347] Connected to a master server at 127.24.255.62:44677
I20260926 05:54:42.490309 9261 heartbeater.cc:464] Registering TS with master...
I20260926 05:54:42.490510 9261 heartbeater.cc:511] Master 127.24.255.62:44677 requested a full tablet report, sending...
I20260926 05:54:42.491364 9071 ts_manager.cc:194] Registered new tserver with Master: 72f33b442e874350a6f59c8bc7a2f9e5 (127.24.255.1:40759)
I20260926 05:54:42.492797 9071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:38853
I20260926 05:54:42.493844 9261 heartbeater.cc:503] Master 127.24.255.62:44677 was elected leader, sending a full tablet report...
Sep 26 05:54:42 dist-test-slave-7fr8 krb5kdc[8491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for kudu/127.24.255.1@KRBTEST.COM
I20260926 05:54:42.501453 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.492132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:40759 (local address 127.0.0.1:40316)
0926 05:54:42.492375 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:42.492410 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:42.492488 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:42.492663 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:42.492667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:42.493008 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:42.493064 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.493067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.495139 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.495140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.495750 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.495753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.496423 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.496424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.496500 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.497098 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:42.497103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:42.498510 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:42.500332 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:42.500339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:42.500341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:42.500545 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:42.500743 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:42.500743 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:42.500743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:42.500794 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:42.501107 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:42.501114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:42.501168 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 05:54:42.501245 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 05:54:42.501355 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":101,"threads_started":1}
I20260926 05:54:42.501518 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.492218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.24.255.1:40759)
0926 05:54:42.492381 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:42.492385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:42.492502 (+ 117us) server_negotiation.cc:407] Connection header received
0926 05:54:42.492576 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:42.492579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:42.492596 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:42.492656 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:42.493165 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.495008 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.495835 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.496327 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.496401 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.498614 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:42.498625 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:42.498626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:42.498634 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:42.500232 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:42.500622 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:42.500625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:42.500625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:42.500688 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:42.500841 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:42.500842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:42.500842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:42.501019 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:42.501105 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:42.501298 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 05:54:42.501404 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260926 05:54:42.502075 9195 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:54:22 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:40316
I20260926 05:54:42.502732 9270 tablet_replica.cc:330] stopping tablet replica
I20260926 05:54:42.502851 9270 raft_consensus.cc:2229] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:54:42.502959 9270 raft_consensus.cc:2258] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:54:42.503288 9270 ts_tablet_manager.cc:1905] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:54:42.505544 9270 ts_tablet_manager.cc:1918] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:54:42.505636 9270 log.cc:1196] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal/wals/a3e3ffcdbf1f49a2a5e72581b0bd50ab
I20260926 05:54:42.505843 9270 ts_tablet_manager.cc:1939] T a3e3ffcdbf1f49a2a5e72581b0bd50ab P 72f33b442e874350a6f59c8bc7a2f9e5: Deleting consensus metadata
I20260926 05:54:42.506421 9057 catalog_manager.cc:4841] TS 72f33b442e874350a6f59c8bc7a2f9e5 (127.24.255.1:40759): tablet a3e3ffcdbf1f49a2a5e72581b0bd50ab (table test-table [id=256f5ea2913d46d69e6f0d3e406ba50c]) successfully deleted
W20260926 05:54:42.580121 9266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:42.580351 9266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:42.580379 9266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:42.581758 9266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:42.581807 9266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:42.581838 9266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:54:42.583544 9266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:41731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=37187
--tserver_master_addrs=127.24.255.62:44677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:42.583773 9266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:42.583968 9266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:54:42.589073 9275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:42.589077 9278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:42.589241 9276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:42.589375 9266 server_base.cc:1034] running on GCE node
I20260926 05:54:42.589740 9266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:42.590441 9266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:42.591691 9266 hybrid_clock.cc:648] HybridClock initialized: now 1790402082591659 us; error 74 us; skew 500 ppm
Sep 26 05:54:42 dist-test-slave-7fr8 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:42.595170 9266 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:54:42.595731 9266 webserver.cc:460] Webserver started at http://127.24.255.2:37187/ using document root <none> and password file <none>
I20260926 05:54:42.595963 9266 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:42.596029 9266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:42.597462 9266 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:42.598137 9285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:42.598300 9266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:42.598531 9266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "f6d3d128fefa4f2297b187f60cb6f2a8"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "67a17c1e7fe95d8ce02b98dc4e99de85"
server_key_iv: "fb4fe315bae14c055e1b598eb448d8bf"
server_key_version: "encryptionkey@0"
I20260926 05:54:42.599252 9266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:42.632874 9266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:42.635762 9266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:42.635946 9266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:42.636207 9266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:42.636518 9266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:54:42.636554 9266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:42.636584 9266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:54:42.636602 9266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:42.642952 9266 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:41731
I20260926 05:54:42.643057 9398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:41731 every 8 connection(s)
I20260926 05:54:42.644755 9266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:54:42.651228 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 9266
I20260926 05:54:42.651702 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:44513
--local_ip_for_outbound_sockets=127.24.255.3
--tserver_master_addrs=127.24.255.62:44677
--webserver_port=35899
--webserver_interface=127.24.255.3
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:54:42 dist-test-slave-7fr8 krb5kdc[8491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:54:42.657306 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.645888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:47029 (local address 127.24.255.62:44677)
0926 05:54:42.645998 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:42.646000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:42.646248 (+ 248us) server_negotiation.cc:407] Connection header received
0926 05:54:42.646784 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:42.646786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:42.646803 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:42.646877 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:42.648300 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.648881 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.650209 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.650450 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.650530 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.653815 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:42.653817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:42.653818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:42.653829 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:42.655538 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:42.656113 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:42.656115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:42.656115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:42.656184 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:42.656538 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:42.656539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:42.656539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:42.656694 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:42.656766 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:42.657094 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 05:54:42.657202 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 05:54:42.657405 9401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.645823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:47029)
0926 05:54:42.646005 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:42.646055 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:42.646620 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:42.647077 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:42.647084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:42.647872 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:42.648136 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.648144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.649031 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.649032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.650034 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.650037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.650556 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.650557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.650676 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.651621 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:42.651635 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:42.653685 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:42.655672 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:42.655682 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:42.655684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:42.656027 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:42.656285 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:42.656285 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:42.656286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:42.656422 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:42.656798 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:42.656805 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:42.656955 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 05:54:42.657052 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 05:54:42.657210 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260926 05:54:42.657891 9399 heartbeater.cc:347] Connected to a master server at 127.24.255.62:44677
I20260926 05:54:42.657982 9399 heartbeater.cc:464] Registering TS with master...
I20260926 05:54:42.658206 9399 heartbeater.cc:511] Master 127.24.255.62:44677 requested a full tablet report, sending...
I20260926 05:54:42.658608 9071 ts_manager.cc:194] Registered new tserver with Master: f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731)
I20260926 05:54:42.658980 9071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:47029
W20260926 05:54:42.742961 9402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:54:42.743196 9402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:54:42.743229 9402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:54:42.744525 9402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:54:42.744589 9402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:54:42.744630 9402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:54:42.746557 9402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:44513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=35899
--tserver_master_addrs=127.24.255.62:44677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:54:42.746840 9402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:54:42.747083 9402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:54:42.753250 9402 server_base.cc:1034] running on GCE node
W20260926 05:54:42.753172 9410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:42.753289 9407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:54:42.753172 9408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:54:42.753722 9402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:54:42.754024 9402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:54:42.755195 9402 hybrid_clock.cc:648] HybridClock initialized: now 1790402082755180 us; error 34 us; skew 500 ppm
Sep 26 05:54:42 dist-test-slave-7fr8 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:54:42.757920 9402 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:54:42.758525 9402 webserver.cc:460] Webserver started at http://127.24.255.3:35899/ using document root <none> and password file <none>
I20260926 05:54:42.758733 9402 fs_manager.cc:359] Metadata directory not provided
I20260926 05:54:42.758797 9402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:54:42.760186 9402 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:54:42.760895 9417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:42.761108 9402 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:54:42.761201 9402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "4a0ea34d1a2540cb969e9b49c28d6596"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "554243cad2f7353cbc516ba67ea646f7"
server_key_iv: "05c3641a35f0036092c10851fa508474"
server_key_version: "encryptionkey@0"
I20260926 05:54:42.761579 9402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:54:42.773590 9402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:54:42.776168 9402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:54:42.776325 9402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:54:42.776702 9402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:54:42.777024 9402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:54:42.777060 9402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:42.777091 9402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:54:42.777112 9402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:54:42.783634 9402 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:44513
I20260926 05:54:42.783689 9530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:44513 every 8 connection(s)
I20260926 05:54:42.784091 9402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:54:42.788272 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 9402
Sep 26 05:54:42 dist-test-slave-7fr8 krb5kdc[8491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:54:42.795259 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.784892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:44211 (local address 127.24.255.62:44677)
0926 05:54:42.784991 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:42.784993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:42.785172 (+ 179us) server_negotiation.cc:407] Connection header received
0926 05:54:42.785657 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:42.785659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:42.785669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:42.785732 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:42.786899 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.787431 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.788606 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.788877 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.788936 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.791634 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:42.791635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:42.791636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:42.791642 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:42.793161 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:42.793703 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:42.793707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:42.793707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:42.793823 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:42.794414 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:42.794417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:42.794417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:42.794652 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:42.794746 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:42.795104 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 05:54:42.795185 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 05:54:42.795492 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.784856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:44211)
0926 05:54:42.785016 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:42.785059 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:42.785545 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:42.785852 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:42.785858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:42.786563 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:54:42.786772 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.786778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.787540 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.787540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.788516 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.788519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.788999 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.789000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.789160 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.789666 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:54:42.789675 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:54:42.791519 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:54:42.793249 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:42.793256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:42.793259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:42.793572 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:42.794011 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:54:42.794012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:54:42.794013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:54:42.794311 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:54:42.794773 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:54:42.794780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:54:42.794950 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 05:54:42.795047 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 05:54:42.795242 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260926 05:54:42.796059 9531 heartbeater.cc:347] Connected to a master server at 127.24.255.62:44677
I20260926 05:54:42.796195 9531 heartbeater.cc:464] Registering TS with master...
I20260926 05:54:42.796444 9531 heartbeater.cc:511] Master 127.24.255.62:44677 requested a full tablet report, sending...
I20260926 05:54:42.796727 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.788695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.24.255.62:44677)
0926 05:54:42.788823 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:42.788825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:42.789015 (+ 190us) server_negotiation.cc:407] Connection header received
0926 05:54:42.789036 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:42.789036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:42.789049 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:42.789121 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:42.789723 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.790490 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.791521 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.791825 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.791874 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.793041 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:42.793043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:42.793043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:42.793050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:42.795050 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:42.795563 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:42.795737 (+ 174us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:42.795737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:42.795792 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:42.796011 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:42.796012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:42.796013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:42.796162 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:42.796333 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:42.796544 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 05:54:42.796639 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260926 05:54:42.796999 9070 ts_manager.cc:194] Registered new tserver with Master: 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513)
I20260926 05:54:42.797458 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:54:42.797592 9070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:44211
I20260926 05:54:42.800348 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.797797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.24.255.62:44677)
0926 05:54:42.797832 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:42.797834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:42.797969 (+ 135us) server_negotiation.cc:407] Connection header received
0926 05:54:42.798081 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:42.798083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:42.798155 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:42.798242 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:54:42.798343 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.798893 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.799721 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.799917 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.799976 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.800145 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:54:42.800186 (+ 41us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 05:54:42.800295 (+ 109us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52336: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 05:54:42.800398 9534 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52336: token expired
W20260926 05:54:42.800987 8998 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.255.62:44677)
I20260926 05:54:42.807884 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.801261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.24.255.62:44677)
0926 05:54:42.801300 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:42.801302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:42.801309 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:54:42.801356 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:42.801357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:42.801365 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:42.801406 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:54:42.801866 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.802392 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.803143 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.803326 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.803379 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.804957 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:54:42.804959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:54:42.804959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:54:42.804967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:54:42.806482 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:42.806948 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:42.806951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:42.806951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:42.807052 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:54:42.807369 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:54:42.807370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:54:42.807371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:54:42.807550 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:54:42.807640 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:42.807737 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 05:54:42.807809 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:54:42.809130 9070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:54:42.810786 9070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:54:42.813625 9195 tablet_service.cc:1467] Processing CreateTablet for tablet b6b5a975bdbf4aa88ba97b94312304dd (DEFAULT_TABLE table=test-table [id=b2f02c74de784ca59a826140c895c7f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:54:42.813843 9195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b5a975bdbf4aa88ba97b94312304dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:42.815527 9153 tablet_bootstrap.cc:492] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5: Bootstrap starting.
I20260926 05:54:42.816157 9153 tablet_bootstrap.cc:654] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:42.817052 9153 tablet_bootstrap.cc:492] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5: No bootstrap required, opened a new log
I20260926 05:54:42.817133 9153 ts_tablet_manager.cc:1397] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:54:42.817514 9153 raft_consensus.cc:348] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:54:42.817606 9153 raft_consensus.cc:374] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:42.817662 9153 raft_consensus.cc:729] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f33b442e874350a6f59c8bc7a2f9e5, State: Initialized, Role: FOLLOWER
I20260926 05:54:42.817705 9153 consensus_queue.cc:260] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:54:42.817798 9153 ts_tablet_manager.cc:1428] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:54:42.817824 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.812992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:44513 (local address 127.0.0.1:60132)
0926 05:54:42.813095 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:42.813111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:42.813184 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:42.813959 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:42.813961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:42.813963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:42.814046 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.814049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.815394 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.815395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.816366 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.816370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.817519 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.817520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.817601 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.817635 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 05:54:42.817686 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:54:42.817742 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 05:54:42.818161 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.813323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.24.255.3:44513)
0926 05:54:42.813573 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:42.813578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:42.813590 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:54:42.813633 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:42.813635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:42.813653 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:42.814162 (+ 509us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:42.814255 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.815281 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.816573 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.817433 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.817507 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.817574 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:42.818005 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 05:54:42.818053 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":134,"threads_started":1}
I20260926 05:54:42.818334 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.813069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:41731 (local address 127.0.0.1:59720)
0926 05:54:42.813297 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:42.813309 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:42.813357 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:42.813773 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:42.813774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:42.813777 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:42.813882 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.813885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.815561 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.815563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.816770 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.816773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.818064 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.818065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.818167 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.818172 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:54:42.818216 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 05:54:42.818268 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20260926 05:54:42.818544 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.813222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.24.255.2:41731)
0926 05:54:42.813472 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:42.813479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:42.813518 (+ 39us) server_negotiation.cc:407] Connection header received
0926 05:54:42.813588 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:42.813591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:42.813608 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:42.813763 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:42.813971 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.815440 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.816869 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.817887 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.817962 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.818030 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:42.818423 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 05:54:42.818471 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":96,"threads_started":1}
I20260926 05:54:42.818890 9465 tablet_service.cc:1467] Processing CreateTablet for tablet b6b5a975bdbf4aa88ba97b94312304dd (DEFAULT_TABLE table=test-table [id=b2f02c74de784ca59a826140c895c7f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:54:42.819175 9465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b5a975bdbf4aa88ba97b94312304dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:42.819144 9333 tablet_service.cc:1467] Processing CreateTablet for tablet b6b5a975bdbf4aa88ba97b94312304dd (DEFAULT_TABLE table=test-table [id=b2f02c74de784ca59a826140c895c7f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:54:42.819471 9333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b5a975bdbf4aa88ba97b94312304dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:54:42.821990 9541 tablet_bootstrap.cc:492] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596: Bootstrap starting.
I20260926 05:54:42.822191 9542 tablet_bootstrap.cc:492] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8: Bootstrap starting.
I20260926 05:54:42.822729 9541 tablet_bootstrap.cc:654] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:42.822836 9542 tablet_bootstrap.cc:654] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8: Neither blocks nor log segments found. Creating new log.
I20260926 05:54:42.823213 9541 log.cc:824] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:42.823300 9542 log.cc:824] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:54:42.824019 9541 tablet_bootstrap.cc:492] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596: No bootstrap required, opened a new log
I20260926 05:54:42.824092 9541 ts_tablet_manager.cc:1397] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:54:42.824220 9542 tablet_bootstrap.cc:492] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8: No bootstrap required, opened a new log
I20260926 05:54:42.824276 9542 ts_tablet_manager.cc:1397] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:54:42.825604 9541 raft_consensus.cc:348] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:54:42.825753 9541 raft_consensus.cc:374] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:42.825786 9541 raft_consensus.cc:729] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a0ea34d1a2540cb969e9b49c28d6596, State: Initialized, Role: FOLLOWER
I20260926 05:54:42.825778 9542 raft_consensus.cc:348] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:54:42.825953 9542 raft_consensus.cc:374] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:54:42.826002 9542 raft_consensus.cc:729] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3d128fefa4f2297b187f60cb6f2a8, State: Initialized, Role: FOLLOWER
I20260926 05:54:42.825935 9541 consensus_queue.cc:260] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:54:42.826256 9542 consensus_queue.cc:260] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:54:42.826406 9531 heartbeater.cc:503] Master 127.24.255.62:44677 was elected leader, sending a full tablet report...
I20260926 05:54:42.826419 9541 ts_tablet_manager.cc:1428] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 05:54:42.826557 9542 ts_tablet_manager.cc:1428] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:54:42.826555 9399 heartbeater.cc:503] Master 127.24.255.62:44677 was elected leader, sending a full tablet report...
I20260926 05:54:42.890672 9546 raft_consensus.cc:482] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:54:42.890823 9546 raft_consensus.cc:504] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:54:42.891271 9546 leader_election.cc:288] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513), 72f33b442e874350a6f59c8bc7a2f9e5 (127.24.255.1:40759)
W20260926 05:54:42.895467 9400 tablet.cc:2369] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:54:42.895586 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.891917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:48025 (local address 127.24.255.1:40759)
0926 05:54:42.891979 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:42.891981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:42.891989 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:54:42.892046 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:42.892048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:42.892079 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:42.892170 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:42.892331 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.893200 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.894441 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.895092 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.895177 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.895263 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:42.895392 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 05:54:42.895487 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:54:42.895915 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.891618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:40759 (local address 127.24.255.2:48025)
0926 05:54:42.891808 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:42.891820 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:42.891881 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:42.892152 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:42.892153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:42.892155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:42.892229 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.892231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.893320 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.893321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.894292 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.894296 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.895220 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.895221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.895303 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.895310 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 05:54:42.895371 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:54:42.895424 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":39,"thread_start_us":94,"threads_started":1}
I20260926 05:54:42.895905 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.891512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:35121 (local address 127.24.255.3:44513)
0926 05:54:42.891558 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:42.891560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:42.891955 (+ 395us) server_negotiation.cc:407] Connection header received
0926 05:54:42.891987 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:42.891988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:42.891998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:42.892048 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:54:42.892534 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.893228 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.894664 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.895214 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.895305 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.895315 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:42.895647 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 05:54:42.895823 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:54:42.896150 9401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.891477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:44513 (local address 127.24.255.2:35121)
0926 05:54:42.891758 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:42.891769 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:54:42.891817 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:54:42.892083 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:54:42.892085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:54:42.892086 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:54:42.892194 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.892197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.893315 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.893316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.894476 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:54:42.894482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.895334 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.895336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:54:42.895432 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.895437 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:54:42.895498 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:54:42.895561 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":27}
I20260926 05:54:42.896543 9215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b5a975bdbf4aa88ba97b94312304dd" candidate_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" is_pre_election: true
I20260926 05:54:42.896708 9485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b5a975bdbf4aa88ba97b94312304dd" candidate_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" is_pre_election: true
I20260926 05:54:42.896806 9215 raft_consensus.cc:2454] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3d128fefa4f2297b187f60cb6f2a8 in term 0.
I20260926 05:54:42.896885 9485 raft_consensus.cc:2454] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3d128fefa4f2297b187f60cb6f2a8 in term 0.
I20260926 05:54:42.897142 9288 leader_election.cc:302] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72f33b442e874350a6f59c8bc7a2f9e5, f6d3d128fefa4f2297b187f60cb6f2a8; no voters:
I20260926 05:54:42.897398 9546 raft_consensus.cc:2781] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:54:42.897500 9546 raft_consensus.cc:482] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:54:42.897524 9546 raft_consensus.cc:3037] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:42.898305 9546 raft_consensus.cc:504] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:54:42.898450 9546 leader_election.cc:288] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [CANDIDATE]: Term 1 election: Requested vote from peers 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513), 72f33b442e874350a6f59c8bc7a2f9e5 (127.24.255.1:40759)
I20260926 05:54:42.898670 9485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b5a975bdbf4aa88ba97b94312304dd" candidate_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a0ea34d1a2540cb969e9b49c28d6596"
I20260926 05:54:42.898679 9215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b5a975bdbf4aa88ba97b94312304dd" candidate_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f33b442e874350a6f59c8bc7a2f9e5"
I20260926 05:54:42.898757 9485 raft_consensus.cc:3037] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:42.898775 9215 raft_consensus.cc:3037] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:54:42.899518 9215 raft_consensus.cc:2454] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3d128fefa4f2297b187f60cb6f2a8 in term 1.
I20260926 05:54:42.899546 9485 raft_consensus.cc:2454] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3d128fefa4f2297b187f60cb6f2a8 in term 1.
I20260926 05:54:42.899734 9288 leader_election.cc:302] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a0ea34d1a2540cb969e9b49c28d6596, f6d3d128fefa4f2297b187f60cb6f2a8; no voters:
I20260926 05:54:42.899844 9546 raft_consensus.cc:2781] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:54:42.900009 9546 raft_consensus.cc:686] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [term 1 LEADER]: Becoming Leader. State: Replica: f6d3d128fefa4f2297b187f60cb6f2a8, State: Running, Role: LEADER
I20260926 05:54:42.900130 9546 consensus_queue.cc:237] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:54:42.901037 9070 catalog_manager.cc:5495] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2). New cstate: current_term: 1 leader_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } health_report { overall_health: UNKNOWN } } }
I20260926 05:54:42.915508 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.912836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59730 (local address 127.24.255.2:41731)
0926 05:54:42.912918 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 05:54:42.912920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:54:42.912930 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:54:42.913023 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:54:42.913025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:54:42.913051 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:54:42.913184 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:54:42.913389 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.913918 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.914757 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:54:42.914942 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:54:42.914999 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:54:42.915149 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:54:42.915259 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:54:42.915333 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:54:42.915371 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 05:54:42.915399 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 05:54:42.917974 9485 raft_consensus.cc:1261] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596 [term 1 FOLLOWER]: Refusing update from remote peer f6d3d128fefa4f2297b187f60cb6f2a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:54:42.917974 9215 raft_consensus.cc:1261] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Refusing update from remote peer f6d3d128fefa4f2297b187f60cb6f2a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:54:42.918402 9546 consensus_queue.cc:1035] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:54:42.918566 9548 consensus_queue.cc:1035] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:54:42.921054 9552 mvcc.cc:205] Tried to move back new op lower bound from 7333486931628904448 to 7333486931559464960. Current Snapshot: MvccSnapshot[applied={T|T < 7333486931628904448}]
I20260926 05:54:42.922005 9553 mvcc.cc:205] Tried to move back new op lower bound from 7333486931628904448 to 7333486931559464960. Current Snapshot: MvccSnapshot[applied={T|T < 7333486931628904448 or (T in {7333486931628904448})}]
I20260926 05:54:42.923225 9071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:54:42.923316 9071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:54:42.924196 9071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Sending DeleteTablet for 3 replicas of tablet b6b5a975bdbf4aa88ba97b94312304dd
I20260926 05:54:42.924528 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 9110
I20260926 05:54:42.924567 9333 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b5a975bdbf4aa88ba97b94312304dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:54:42 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:59720
I20260926 05:54:42.924608 9195 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b5a975bdbf4aa88ba97b94312304dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:54:42 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:40316
I20260926 05:54:42.924785 9465 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b5a975bdbf4aa88ba97b94312304dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:54:42 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:60132
I20260926 05:54:42.925253 9561 tablet_replica.cc:330] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596: stopping tablet replica
I20260926 05:54:42.925364 9561 raft_consensus.cc:2229] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:54:42.925486 9561 raft_consensus.cc:2258] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:54:42.925740 9560 tablet_replica.cc:330] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8: stopping tablet replica
I20260926 05:54:42.925840 9560 raft_consensus.cc:2229] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:54:42.925941 9560 raft_consensus.cc:2258] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:54:42.926414 9561 ts_tablet_manager.cc:1905] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:54:42.926476 9560 ts_tablet_manager.cc:1905] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:54:42.928359 9560 ts_tablet_manager.cc:1918] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:54:42.928359 9561 ts_tablet_manager.cc:1918] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:54:42.928449 9561 log.cc:1196] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/b6b5a975bdbf4aa88ba97b94312304dd
I20260926 05:54:42.928449 9560 log.cc:1196] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/b6b5a975bdbf4aa88ba97b94312304dd
I20260926 05:54:42.928746 9560 ts_tablet_manager.cc:1939] T b6b5a975bdbf4aa88ba97b94312304dd P f6d3d128fefa4f2297b187f60cb6f2a8: Deleting consensus metadata
I20260926 05:54:42.928771 9561 ts_tablet_manager.cc:1939] T b6b5a975bdbf4aa88ba97b94312304dd P 4a0ea34d1a2540cb969e9b49c28d6596: Deleting consensus metadata
I20260926 05:54:42.929347 9057 catalog_manager.cc:4841] TS 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513): tablet b6b5a975bdbf4aa88ba97b94312304dd (table test-table [id=b2f02c74de784ca59a826140c895c7f0]) successfully deleted
I20260926 05:54:42.929438 9057 catalog_manager.cc:4841] TS f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731): tablet b6b5a975bdbf4aa88ba97b94312304dd (table test-table [id=b2f02c74de784ca59a826140c895c7f0]) successfully deleted
W20260926 05:54:42.929876 9057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:40759 (error 108)
I20260926 05:54:42.930176 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 9266
I20260926 05:54:42.930871 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:54:42.930279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:40759 (local address 127.0.0.1:40330)
0926 05:54:42.930811 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 05:54:42.930843 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:40759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512}
W20260926 05:54:42.931849 9057 catalog_manager.cc:4584] TS 72f33b442e874350a6f59c8bc7a2f9e5 (127.24.255.1:40759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b6b5a975bdbf4aa88ba97b94312304dd: Network error: Client connection negotiation failed: client connection to 127.24.255.1:40759: connect: Connection refused (error 111)
I20260926 05:54:42.935527 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 9402
I20260926 05:54:42.940428 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 9038
I20260926 05:55:02.945793 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:44677
--webserver_interface=127.24.255.62
--webserver_port=36547
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:44677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:03.034461 9563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:03.034688 9563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:03.034713 9563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:03.036126 9563 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 05:55:03.036190 9563 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 05:55:03.036202 9563 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 05:55:03.036213 9563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:03.036226 9563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:03.036238 9563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:03.036248 9563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:03.036259 9563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:03.037966 9563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:44677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:44677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:03.038259 9563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:03.038515 9563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:03.043802 9569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:03.043807 9568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:03.043947 9563 server_base.cc:1034] running on GCE node
W20260926 05:55:03.043802 9571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:03.044441 9563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:03.044775 9563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:03.045964 9563 hybrid_clock.cc:648] HybridClock initialized: now 1790402103045955 us; error 37 us; skew 500 ppm
Sep 26 05:55:03 dist-test-slave-7fr8 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402103, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:03.048763 9563 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:55:03.049242 9563 webserver.cc:460] Webserver started at http://127.24.255.62:36547/ using document root <none> and password file <none>
I20260926 05:55:03.049470 9563 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:03.049522 9563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:03.051020 9563 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:03.051728 9578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:03.051898 9563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:55:03.051983 9563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "7588e6e0324946cd9fb04fa8c7446da9"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "5a9feea4f20bb2a423319392fd44f8fc"
server_key_iv: "9e8febdab757184dbad539d59657d172"
server_key_version: "encryptionkey@0"
I20260926 05:55:03.052361 9563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:03.069283 9563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:03.071766 9563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:03.071916 9563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:03.075821 9563 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:44677
I20260926 05:55:03.076403 9563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:03.076870 9630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:44677 every 8 connection(s)
I20260926 05:55:03.078430 9631 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:55:03.079242 9631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Bootstrap starting.
I20260926 05:55:03.081725 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 9563
I20260926 05:55:03.082044 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:40759
--local_ip_for_outbound_sockets=127.24.255.1
--tserver_master_addrs=127.24.255.62:44677
--webserver_port=42819
--webserver_interface=127.24.255.1
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:55:03.088524 9631 log.cc:824] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:03.090888 9631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:55:03.091188 9631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Bootstrap complete.
I20260926 05:55:03.093299 9631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } }
I20260926 05:55:03.093549 9631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7588e6e0324946cd9fb04fa8c7446da9, State: Initialized, Role: FOLLOWER
I20260926 05:55:03.093708 9631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } }
I20260926 05:55:03.093787 9631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:03.093838 9631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:03.093885 9631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 2 FOLLOWER]: Advancing to term 3
I20260926 05:55:03.095062 9631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } }
I20260926 05:55:03.095172 9631 leader_election.cc:302] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7588e6e0324946cd9fb04fa8c7446da9; no voters:
I20260926 05:55:03.095348 9631 leader_election.cc:288] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 05:55:03.095404 9635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 05:55:03.095528 9635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [term 3 LEADER]: Becoming Leader. State: Replica: 7588e6e0324946cd9fb04fa8c7446da9, State: Running, Role: LEADER
I20260926 05:55:03.095660 9631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:03.095816 9635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } }
I20260926 05:55:03.096321 9636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7588e6e0324946cd9fb04fa8c7446da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } } }
I20260926 05:55:03.096479 9636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:03.096938 9644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:03.097196 9637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7588e6e0324946cd9fb04fa8c7446da9. Latest consensus state: current_term: 3 leader_uuid: "7588e6e0324946cd9fb04fa8c7446da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7588e6e0324946cd9fb04fa8c7446da9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 44677 } } }
I20260926 05:55:03.097308 9637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:03.097875 9644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:03.097994 9644 catalog_manager.cc:1259] Loaded cluster ID: 1361df9a144345c3b63c2bca5bf48c21
I20260926 05:55:03.098029 9644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:03.099191 9644 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:03.099397 9644 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Loaded TSK: 0
I20260926 05:55:03.099460 9644 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7588e6e0324946cd9fb04fa8c7446da9: Loaded TSK: 1
F20260926 05:55:03.099695 9644 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efe824a49ad google::LogMessage::Fail() at ??:0
@ 0x7efe824a788a google::LogMessage::SendToLog() at ??:0
@ 0x7efe824a44de google::LogMessage::Flush() at ??:0
@ 0x7efe824a5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efe84c11104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7efe84c0ed41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7efe8a3fe1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7efe8a3fe48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7efe8a3fe594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7efe8a3c4c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7efe8a3cead0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efe843720d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efe84367954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7efe851cb6db start_thread at ??:0
@ 0x7efe80fa261f clone at ??:0
W20260926 05:55:03.187479 9633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:03.187711 9633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:03.187738 9633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:03.189011 9633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:03.189065 9633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:03.189096 9633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:55:03.191244 9633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:40759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=42819
--tserver_master_addrs=127.24.255.62:44677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:03.191768 9633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:03.192132 9633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:03.199131 9675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:03.199234 9633 server_base.cc:1034] running on GCE node
W20260926 05:55:03.199164 9673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:03.199366 9672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:03.199610 9633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:03.199867 9633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:03.201057 9633 hybrid_clock.cc:648] HybridClock initialized: now 1790402103201040 us; error 29 us; skew 500 ppm
Sep 26 05:55:03 dist-test-slave-7fr8 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402103, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:03.203747 9633 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:55:03.204244 9633 webserver.cc:460] Webserver started at http://127.24.255.1:42819/ using document root <none> and password file <none>
I20260926 05:55:03.204461 9633 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:03.204514 9633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:03.206374 9633 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:03.207497 9682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:03.207753 9633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:03.207870 9633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "72f33b442e874350a6f59c8bc7a2f9e5"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "125a7e54446c22ff7ad2259e2839390e"
server_key_iv: "a3854e50c16a9a34b6d3629af8220749"
server_key_version: "encryptionkey@0"
I20260926 05:55:03.208305 9633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:03.238922 9633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:03.241600 9633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:03.242316 9633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:03.242730 9633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:03.243319 9689 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 05:55:03.245931 9633 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 05:55:03.246022 9633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260926 05:55:03.246063 9633 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 05:55:03.246801 9633 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 05:55:03.246840 9633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 05:55:03.246927 9689 tablet_bootstrap.cc:492] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5: Bootstrap starting.
I20260926 05:55:03.255281 9633 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:40759
I20260926 05:55:03.255872 9633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:55:03.256596 9796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:40759 every 8 connection(s)
I20260926 05:55:03.258703 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 9633
I20260926 05:55:03.259073 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:41731
--local_ip_for_outbound_sockets=127.24.255.2
--tserver_master_addrs=127.24.255.62:44677
--webserver_port=37187
--webserver_interface=127.24.255.2
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:55:03.261750 9689 log.cc:824] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:03.265146 9689 tablet_bootstrap.cc:492] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:55:03.265507 9689 tablet_bootstrap.cc:492] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5: Bootstrap complete.
I20260926 05:55:03.266635 9689 ts_tablet_manager.cc:1397] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.018s
I20260926 05:55:03.268265 9689 raft_consensus.cc:348] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:55:03.268642 9689 raft_consensus.cc:729] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f33b442e874350a6f59c8bc7a2f9e5, State: Initialized, Role: FOLLOWER
I20260926 05:55:03.268844 9689 consensus_queue.cc:260] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:55:03.269178 9689 ts_tablet_manager.cc:1428] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260926 05:55:03.320003 9799 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:03.256668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:51795)
0926 05:55:03.256914 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:03.257061 (+ 147us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:03.257818 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:03.319798 (+ 61980us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
W20260926 05:55:03.320240 9797 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:44677 (0 consecutive failures): Network error: Failed to ping master at 127.24.255.62:44677: Client connection negotiation failed: client connection to 127.24.255.62:44677: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 05:55:03.320600 9799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:03.320482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:38109)
0926 05:55:03.320536 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:03.320576 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:55:03.320966 9799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:03.320873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:51067)
0926 05:55:03.320922 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:03.320946 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 05:55:03.321077 9797 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:55:03.351675 9800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:03.351908 9800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:03.351944 9800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:03.353227 9800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:03.353281 9800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:03.353322 9800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:55:03.355664 9800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:41731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=37187
--tserver_master_addrs=127.24.255.62:44677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:03.356027 9800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:03.356309 9800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:03.361603 9808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:03.361651 9810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:03.361603 9807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:03.361650 9800 server_base.cc:1034] running on GCE node
I20260926 05:55:03.362203 9800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:03.362465 9800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:03.363642 9800 hybrid_clock.cc:648] HybridClock initialized: now 1790402103363625 us; error 32 us; skew 500 ppm
Sep 26 05:55:03 dist-test-slave-7fr8 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402103, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:03.366530 9800 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:55:03.366963 9800 webserver.cc:460] Webserver started at http://127.24.255.2:37187/ using document root <none> and password file <none>
I20260926 05:55:03.367185 9800 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:03.367233 9800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:03.368673 9800 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:03.369519 9817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:03.369753 9800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:03.369846 9800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "f6d3d128fefa4f2297b187f60cb6f2a8"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "67a17c1e7fe95d8ce02b98dc4e99de85"
server_key_iv: "fb4fe315bae14c055e1b598eb448d8bf"
server_key_version: "encryptionkey@0"
I20260926 05:55:03.370275 9800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:03.379458 9800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:03.381731 9800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:03.381873 9800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:03.382159 9800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:03.382462 9800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:03.382495 9800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:03.382525 9800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:03.382550 9800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:03.389441 9800 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:41731
I20260926 05:55:03.389493 9930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:41731 every 8 connection(s)
I20260926 05:55:03.389849 9800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:55:03.391067 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:03.390570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:33673)
0926 05:55:03.390848 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:03.390922 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":73,"threads_started":1}
W20260926 05:55:03.391484 9931 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:44677 (0 consecutive failures): Network error: Failed to ping master at 127.24.255.62:44677: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
I20260926 05:55:03.391788 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:03.391675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:47375)
0926 05:55:03.391735 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:03.391765 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 05:55:03.392038 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:03.391969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:46563)
0926 05:55:03.392002 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:03.392023 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 05:55:03.392167 9931 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:55:03.396534 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 9800
I20260926 05:55:03.396775 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:44513
--local_ip_for_outbound_sockets=127.24.255.3
--tserver_master_addrs=127.24.255.62:44677
--webserver_port=35899
--webserver_interface=127.24.255.3
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:03.486621 9934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:03.486850 9934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:03.486876 9934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:03.488163 9934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:03.488224 9934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:03.488252 9934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:55:03.489899 9934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:44513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=35899
--tserver_master_addrs=127.24.255.62:44677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:03.490203 9934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:03.490458 9934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:03.496132 9942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:03.496184 9934 server_base.cc:1034] running on GCE node
W20260926 05:55:03.496112 9940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:03.496114 9939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:03.496627 9934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:03.496894 9934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:03.498050 9934 hybrid_clock.cc:648] HybridClock initialized: now 1790402103498003 us; error 71 us; skew 500 ppm
Sep 26 05:55:03 dist-test-slave-7fr8 krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402103, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:03.500653 9934 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:55:03.501236 9934 webserver.cc:460] Webserver started at http://127.24.255.3:35899/ using document root <none> and password file <none>
I20260926 05:55:03.501596 9934 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:03.501660 9934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:03.503757 9934 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:03.504647 9949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:03.504848 9934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:03.504941 9934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "4a0ea34d1a2540cb969e9b49c28d6596"
format_stamp: "Formatted at 2026-09-26 05:54:21 on dist-test-slave-7fr8"
server_key: "554243cad2f7353cbc516ba67ea646f7"
server_key_iv: "05c3641a35f0036092c10851fa508474"
server_key_version: "encryptionkey@0"
I20260926 05:55:03.505368 9934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:03.514633 9934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:03.517927 9934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:03.518062 9934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:03.518316 9934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:03.518610 9934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:03.518644 9934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:03.518674 9934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:03.518690 9934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:03.525161 9934 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:44513
I20260926 05:55:03.525264 10062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:44513 every 8 connection(s)
I20260926 05:55:03.525777 9934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:55:03.527374 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:03.526837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:57707)
0926 05:55:03.527105 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:03.527208 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":122,"threads_started":1}
W20260926 05:55:03.528002 10063 heartbeater.cc:644] Failed to heartbeat to 127.24.255.62:44677 (0 consecutive failures): Network error: Failed to ping master at 127.24.255.62:44677: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
I20260926 05:55:03.528508 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:03.528335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:33111)
0926 05:55:03.528441 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:03.528479 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 05:55:03.528975 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:03.528858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:42361)
0926 05:55:03.528921 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:03.528952 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 05:55:03.529165 10063 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:55:03.532593 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 9934
I20260926 05:55:04.322757 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:04.322412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:36625)
0926 05:55:04.322639 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:04.322700 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260926 05:55:04.392799 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:04.392541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:60281)
0926 05:55:04.392698 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:04.392739 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260926 05:55:04.470664 10069 raft_consensus.cc:482] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:55:04.470790 10069 raft_consensus.cc:504] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:55:04.471149 10069 leader_election.cc:288] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513), f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731)
Sep 26 05:55:04 dist-test-slave-7fr8 krb5kdc[8491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402103, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.3@KRBTEST.COM
Sep 26 05:55:04 dist-test-slave-7fr8 krb5kdc[8491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402103, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.2@KRBTEST.COM
I20260926 05:55:04.481690 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:04.471316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:44513 (local address 127.24.255.1:60869)
0926 05:55:04.471398 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:04.471409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:04.471468 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:04.472395 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:04.472398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:04.472788 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:04.473004 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:04.473011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:04.474378 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:04.474379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:04.475342 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:04.475346 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:04.475960 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:04.475961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:04.476016 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:04.476565 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:04.476574 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:04.478144 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:04.480127 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:04.480136 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:04.480138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:04.480461 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:04.480696 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:04.480697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:04.480698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:04.480809 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:04.481183 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:04.481190 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:04.481348 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 05:55:04.481438 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 05:55:04.481614 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":12}
I20260926 05:55:04.481804 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:04.471539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:33653 (local address 127.24.255.2:41731)
0926 05:55:04.471684 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:04.471694 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:04.471721 (+ 27us) server_negotiation.cc:407] Connection header received
0926 05:55:04.472168 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:04.472170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:04.472208 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:04.472341 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:04.473156 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:04.474086 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:04.475348 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:04.475788 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:04.475890 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:04.478570 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:04.478574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:04.478574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:04.478587 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:04.480171 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:04.480555 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:04.480558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:04.480558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:04.480643 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:04.480872 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:04.480873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:04.480874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:04.481119 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:04.481190 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:04.481571 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 05:55:04.481695 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260926 05:55:04.482036 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:04.471413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:60869 (local address 127.24.255.3:44513)
0926 05:55:04.471691 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:04.471700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:04.471720 (+ 20us) server_negotiation.cc:407] Connection header received
0926 05:55:04.472139 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:04.472141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:04.472197 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:04.472325 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:04.473176 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:04.474236 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:04.475473 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:04.475883 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:04.475991 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:04.478287 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:04.478292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:04.478292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:04.478303 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:04.480040 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:04.480546 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:04.480551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:04.480551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:04.480638 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:04.480864 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:04.480865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:04.480866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:04.481120 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:04.481190 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:04.481790 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 05:55:04.481936 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":165,"threads_started":1}
I20260926 05:55:04.481820 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:04.471446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:41731 (local address 127.24.255.1:33653)
0926 05:55:04.471541 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:04.471550 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:04.471592 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:04.472430 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:04.472431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:04.472788 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:04.473004 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:04.473011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:04.474221 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:04.474222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:04.475273 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:04.475276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:04.475874 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:04.475875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:04.475986 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:04.476567 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:04.476576 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:04.478451 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:04.480269 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:04.480270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:04.480270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:04.480481 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:04.480703 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:04.480704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:04.480704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:04.480815 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:04.481183 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:04.481190 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:04.481332 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 05:55:04.481501 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 05:55:04.481694 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":107,"thread_start_us":50,"threads_started":1}
W20260926 05:55:04.483206 9685 leader_election.cc:341] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513): Not found: Tablet not found: b6b5a975bdbf4aa88ba97b94312304dd
W20260926 05:55:04.483318 9685 leader_election.cc:341] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731): Not found: Tablet not found: b6b5a975bdbf4aa88ba97b94312304dd
I20260926 05:55:04.483345 9685 leader_election.cc:302] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72f33b442e874350a6f59c8bc7a2f9e5; no voters: 4a0ea34d1a2540cb969e9b49c28d6596, f6d3d128fefa4f2297b187f60cb6f2a8
I20260926 05:55:04.483458 10069 raft_consensus.cc:2726] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:55:04.529721 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:04.529498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:54357)
0926 05:55:04.529633 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:04.529678 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260926 05:55:05.323621 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:05.323359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:56841)
0926 05:55:05.323522 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:05.323570 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":68,"threads_started":1}
I20260926 05:55:05.393563 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:05.393322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:50427)
0926 05:55:05.393475 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:05.393513 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260926 05:55:05.530514 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:05.530259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:59599)
0926 05:55:05.530402 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:05.530450 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260926 05:55:06.220762 10077 raft_consensus.cc:482] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:55:06.220878 10077 raft_consensus.cc:504] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:55:06.221017 10077 leader_election.cc:288] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513), f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731)
W20260926 05:55:06.221699 9685 leader_election.cc:341] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513): Not found: Tablet not found: b6b5a975bdbf4aa88ba97b94312304dd
W20260926 05:55:06.221800 9685 leader_election.cc:341] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731): Not found: Tablet not found: b6b5a975bdbf4aa88ba97b94312304dd
I20260926 05:55:06.221817 9685 leader_election.cc:302] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72f33b442e874350a6f59c8bc7a2f9e5; no voters: 4a0ea34d1a2540cb969e9b49c28d6596, f6d3d128fefa4f2297b187f60cb6f2a8
I20260926 05:55:06.221903 10077 raft_consensus.cc:2726] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:55:06.324517 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:06.324261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:54117)
0926 05:55:06.324418 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:06.324466 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260926 05:55:06.394562 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:06.394245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:33509)
0926 05:55:06.394414 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:06.394462 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260926 05:55:06.531564 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:06.531198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:33239)
0926 05:55:06.531440 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:06.531514 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":75,"threads_started":1}
I20260926 05:55:07.325482 10081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:07.325216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:51403)
0926 05:55:07.325385 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:07.325433 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260926 05:55:07.395560 10082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:07.395230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:59001)
0926 05:55:07.395458 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:07.395510 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":69,"threads_started":1}
I20260926 05:55:07.532563 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:07.532250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:40767)
0926 05:55:07.532416 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:07.532516 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260926 05:55:08.326683 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:08.326320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:49471)
0926 05:55:08.326535 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:08.326614 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":83,"threads_started":1}
I20260926 05:55:08.396589 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:08.396346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:39063)
0926 05:55:08.396504 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:08.396539 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260926 05:55:08.533401 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:08.533132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:50527)
0926 05:55:08.533303 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:08.533351 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":67,"threads_started":1}
I20260926 05:55:09.327559 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:09.327279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:46869)
0926 05:55:09.327461 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:09.327509 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260926 05:55:09.395454 10088 raft_consensus.cc:482] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:55:09.395565 10088 raft_consensus.cc:504] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:55:09.395704 10088 leader_election.cc:288] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513), f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731)
W20260926 05:55:09.396226 9685 leader_election.cc:341] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731): Not found: Tablet not found: b6b5a975bdbf4aa88ba97b94312304dd
W20260926 05:55:09.396309 9685 leader_election.cc:341] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513): Not found: Tablet not found: b6b5a975bdbf4aa88ba97b94312304dd
I20260926 05:55:09.396327 9685 leader_election.cc:302] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72f33b442e874350a6f59c8bc7a2f9e5; no voters: 4a0ea34d1a2540cb969e9b49c28d6596, f6d3d128fefa4f2297b187f60cb6f2a8
I20260926 05:55:09.396394 10088 raft_consensus.cc:2726] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:55:09.397337 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:09.397101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:41043)
0926 05:55:09.397253 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:09.397295 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260926 05:55:09.534209 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:09.533926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:42355)
0926 05:55:09.534083 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:09.534151 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260926 05:55:10.328354 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:10.328093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:53329)
0926 05:55:10.328266 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:10.328308 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260926 05:55:10.398254 10092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:10.397952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:35669)
0926 05:55:10.398154 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:10.398205 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":67,"threads_started":1}
I20260926 05:55:10.535033 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:10.534716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:46233)
0926 05:55:10.534863 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:10.534980 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260926 05:55:11.221746 10094 raft_consensus.cc:482] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:55:11.221840 10094 raft_consensus.cc:504] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:55:11.221974 10094 leader_election.cc:288] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513), f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731)
W20260926 05:55:11.222440 9685 leader_election.cc:341] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731): Not found: Tablet not found: b6b5a975bdbf4aa88ba97b94312304dd
W20260926 05:55:11.222506 9685 leader_election.cc:341] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513): Not found: Tablet not found: b6b5a975bdbf4aa88ba97b94312304dd
I20260926 05:55:11.222523 9685 leader_election.cc:302] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72f33b442e874350a6f59c8bc7a2f9e5; no voters: 4a0ea34d1a2540cb969e9b49c28d6596, f6d3d128fefa4f2297b187f60cb6f2a8
I20260926 05:55:11.222604 10094 raft_consensus.cc:2726] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:55:11.329216 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:11.328951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:34585)
0926 05:55:11.329124 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:11.329168 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
I20260926 05:55:11.399132 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:11.398733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:35267)
0926 05:55:11.398962 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:11.399037 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":71,"threads_started":1}
I20260926 05:55:11.535778 10097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:11.535552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:41933)
0926 05:55:11.535692 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:11.535729 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260926 05:55:12.330400 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:12.330031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:55189)
0926 05:55:12.330284 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:12.330349 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":130,"threads_started":1}
I20260926 05:55:12.400215 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:12.399907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:34091)
0926 05:55:12.400117 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:12.400163 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":93,"threads_started":1}
I20260926 05:55:12.536564 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:12.536316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:41091)
0926 05:55:12.536476 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:12.536517 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":60,"threads_started":1}
I20260926 05:55:13.331336 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:13.331063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:51583)
0926 05:55:13.331237 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:13.331288 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
I20260926 05:55:13.401151 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:13.400857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:58513)
0926 05:55:13.401055 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:13.401103 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":64,"threads_started":1}
I20260926 05:55:13.537490 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:13.537153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:41667)
0926 05:55:13.537344 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:13.537393 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
I20260926 05:55:14.332122 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:14.331867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:51551)
0926 05:55:14.332019 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:14.332075 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260926 05:55:14.402043 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:14.401767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:51657)
0926 05:55:14.401943 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:14.401992 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
I20260926 05:55:14.538357 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:14.538063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:35949)
0926 05:55:14.538242 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:14.538296 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260926 05:55:15.332814 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:15.332585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:58109)
0926 05:55:15.332729 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:15.332769 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260926 05:55:15.402951 10108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:15.402691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:46905)
0926 05:55:15.402853 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:15.402900 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260926 05:55:15.539180 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:15.538912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:54857)
0926 05:55:15.539077 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:15.539124 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260926 05:55:16.333670 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:16.333403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:53811)
0926 05:55:16.333571 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:16.333622 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20260926 05:55:16.403959 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:16.403584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:46459)
0926 05:55:16.403800 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:16.403841 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260926 05:55:16.539964 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:16.539692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:46099)
0926 05:55:16.539862 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:16.539903 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
I20260926 05:55:16.766340 10113 raft_consensus.cc:482] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:55:16.766445 10113 raft_consensus.cc:504] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ea34d1a2540cb969e9b49c28d6596" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 44513 } } peers { permanent_uuid: "f6d3d128fefa4f2297b187f60cb6f2a8" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41731 } } peers { permanent_uuid: "72f33b442e874350a6f59c8bc7a2f9e5" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 40759 } }
I20260926 05:55:16.766636 10113 leader_election.cc:288] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513), f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731)
W20260926 05:55:16.767043 9685 leader_election.cc:341] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a0ea34d1a2540cb969e9b49c28d6596 (127.24.255.3:44513): Not found: Tablet not found: b6b5a975bdbf4aa88ba97b94312304dd
W20260926 05:55:16.767102 9685 leader_election.cc:341] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6d3d128fefa4f2297b187f60cb6f2a8 (127.24.255.2:41731): Not found: Tablet not found: b6b5a975bdbf4aa88ba97b94312304dd
I20260926 05:55:16.767115 9685 leader_election.cc:302] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72f33b442e874350a6f59c8bc7a2f9e5; no voters: 4a0ea34d1a2540cb969e9b49c28d6596, f6d3d128fefa4f2297b187f60cb6f2a8
I20260926 05:55:16.767176 10113 raft_consensus.cc:2726] T b6b5a975bdbf4aa88ba97b94312304dd P 72f33b442e874350a6f59c8bc7a2f9e5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:55:17.334615 10114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:17.334324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:42773)
0926 05:55:17.334513 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:17.334564 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":66,"threads_started":1}
I20260926 05:55:17.404842 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:17.404542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:60511)
0926 05:55:17.404743 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:17.404791 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
I20260926 05:55:17.540763 10116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:17.540486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:59843)
0926 05:55:17.540671 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:17.540715 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260926 05:55:18.335469 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:18.335254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.1:53427)
0926 05:55:18.335380 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:18.335422 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260926 05:55:18.405977 10118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:18.405699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.2:35907)
0926 05:55:18.405872 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:18.405927 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":73,"threads_started":1}
I20260926 05:55:18.538322 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 9633
I20260926 05:55:18.541621 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:18.541381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:44677 (local address 127.24.255.3:38099)
0926 05:55:18.541525 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:18.541567 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.62:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260926 05:55:18.542929 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 9800
I20260926 05:55:18.547154 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 9934
2026-09-26T05:55:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58329 ms)
[----------] 30 tests from SecurityITest (237383 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:18 dist-test-slave-7fr8 krb5kdc[10122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:18 dist-test-slave-7fr8 krb5kdc[10122](info): set up 2 sockets
Sep 26 05:55:18 dist-test-slave-7fr8 krb5kdc[10122](info): commencing operation
krb5kdc: starting...
W20260926 05:55:19.526017 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:19 dist-test-slave-7fr8 krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:55:19.655589 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:35731
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:45255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:35731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:19.738940 10135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:19.739173 10135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:19.739198 10135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:19.740525 10135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:19.740576 10135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:19.740588 10135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:19.740600 10135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:19.740612 10135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:19.742432 10135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:45255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:35731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:35731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:19.742755 10135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:19.743001 10135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:19.748685 10141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:19.748752 10135 server_base.cc:1034] running on GCE node
W20260926 05:55:19.748685 10140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:19.748704 10143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:19.749313 10135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:19.749662 10135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:19.750815 10135 hybrid_clock.cc:648] HybridClock initialized: now 1790402119750786 us; error 48 us; skew 500 ppm
Sep 26 05:55:19 dist-test-slave-7fr8 krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402119, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:19.753430 10135 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:55:19.753885 10135 webserver.cc:460] Webserver started at http://127.24.255.62:36491/ using document root <none> and password file <none>
I20260926 05:55:19.754118 10135 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:19.754179 10135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:19.754310 10135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:19.755375 10135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "a152d1af0b2b405f9196e9e7c865b3d3"
format_stamp: "Formatted at 2026-09-26 05:55:19 on dist-test-slave-7fr8"
server_key: "cbda3f68ae029abe5501cf1f65eab728"
server_key_iv: "26835e70a9f24531538642dc69a37f65"
server_key_version: "encryptionkey@0"
I20260926 05:55:19.755709 10135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "a152d1af0b2b405f9196e9e7c865b3d3"
format_stamp: "Formatted at 2026-09-26 05:55:19 on dist-test-slave-7fr8"
server_key: "cbda3f68ae029abe5501cf1f65eab728"
server_key_iv: "26835e70a9f24531538642dc69a37f65"
server_key_version: "encryptionkey@0"
I20260926 05:55:19.756991 10135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:55:19.757762 10150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:19.757984 10135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:19.758080 10135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "a152d1af0b2b405f9196e9e7c865b3d3"
format_stamp: "Formatted at 2026-09-26 05:55:19 on dist-test-slave-7fr8"
server_key: "cbda3f68ae029abe5501cf1f65eab728"
server_key_iv: "26835e70a9f24531538642dc69a37f65"
server_key_version: "encryptionkey@0"
I20260926 05:55:19.758188 10135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:19.769523 10135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:19.771561 10135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:19.771708 10135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:19.775571 10135 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:35731
I20260926 05:55:19.775614 10202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:35731 every 8 connection(s)
I20260926 05:55:19.775954 10135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:19.776830 10203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:19.779249 10203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3: Bootstrap starting.
I20260926 05:55:19.779841 10203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:19.780093 10203 log.cc:824] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:19.780776 10203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3: No bootstrap required, opened a new log
I20260926 05:55:19.781014 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 10135
I20260926 05:55:19.781112 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:19.781302 25596 external_mini_cluster.cc:1428] Setting key e1f015428428b0947f2be5354fc09d02
I20260926 05:55:19.782825 10203 raft_consensus.cc:348] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a152d1af0b2b405f9196e9e7c865b3d3" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35731 } }
I20260926 05:55:19.782977 10203 raft_consensus.cc:374] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:19.783010 10203 raft_consensus.cc:729] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a152d1af0b2b405f9196e9e7c865b3d3, State: Initialized, Role: FOLLOWER
I20260926 05:55:19.783179 10203 consensus_queue.cc:260] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a152d1af0b2b405f9196e9e7c865b3d3" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35731 } }
I20260926 05:55:19.783269 10203 raft_consensus.cc:388] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:19.783303 10203 raft_consensus.cc:482] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:19.783341 10203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:19.784345 10203 raft_consensus.cc:504] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a152d1af0b2b405f9196e9e7c865b3d3" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35731 } }
I20260926 05:55:19.784440 10203 leader_election.cc:302] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a152d1af0b2b405f9196e9e7c865b3d3; no voters:
I20260926 05:55:19.784592 10203 leader_election.cc:288] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:19.784679 10208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:19.784848 10203 sys_catalog.cc:564] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:19.784902 10208 raft_consensus.cc:686] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [term 1 LEADER]: Becoming Leader. State: Replica: a152d1af0b2b405f9196e9e7c865b3d3, State: Running, Role: LEADER
I20260926 05:55:19.785012 10208 consensus_queue.cc:237] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a152d1af0b2b405f9196e9e7c865b3d3" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35731 } }
I20260926 05:55:19.785454 10208 sys_catalog.cc:455] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a152d1af0b2b405f9196e9e7c865b3d3. Latest consensus state: current_term: 1 leader_uuid: "a152d1af0b2b405f9196e9e7c865b3d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a152d1af0b2b405f9196e9e7c865b3d3" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35731 } } }
I20260926 05:55:19.785612 10208 sys_catalog.cc:458] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:19.785960 10217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:19.786484 10209 sys_catalog.cc:455] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a152d1af0b2b405f9196e9e7c865b3d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a152d1af0b2b405f9196e9e7c865b3d3" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 35731 } } }
I20260926 05:55:19.786568 10217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:19.786652 10209 sys_catalog.cc:458] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:19.788605 10217 catalog_manager.cc:1347] Generated new cluster ID: c3a4f3b5dc4142149ceb214e0d56809d
I20260926 05:55:19.788662 10217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:55:19 dist-test-slave-7fr8 krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:19.792330 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:19.781845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.24.255.62:35731)
0926 05:55:19.782022 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:19.782032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:19.782060 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:55:19.782794 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:19.782802 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:19.782860 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:19.783034 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:19.783761 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:19.784802 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:19.785865 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:19.786524 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:19.786640 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:19.789343 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:19.789348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:19.789348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:19.789361 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:19.790896 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:19.791222 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:19.791225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:19.791225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:19.791324 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:19.791502 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:19.791503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:19.791503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:19.791751 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:19.791884 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:19.791995 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 05:55:19.792144 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260926 05:55:19.812378 10217 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:19.813088 10217 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:19.818624 10217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3: Generated new TSK 0
I20260926 05:55:19.818838 10217 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:55:19.853796 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35731
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:45255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:19.942292 10229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:19.942529 10229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:19.942565 10229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:19.943780 10229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:19.943826 10229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:19.943866 10229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:55:19.945488 10229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:45255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:19.945744 10229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:19.945962 10229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:19.951489 10235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:19.951495 10229 server_base.cc:1034] running on GCE node
W20260926 05:55:19.951503 10234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:19.951512 10237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:19.951915 10229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:19.952160 10229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:19.953334 10229 hybrid_clock.cc:648] HybridClock initialized: now 1790402119953317 us; error 30 us; skew 500 ppm
Sep 26 05:55:19 dist-test-slave-7fr8 krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402119, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:19.956073 10229 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:55:19.956513 10229 webserver.cc:460] Webserver started at http://127.24.255.1:37599/ using document root <none> and password file <none>
I20260926 05:55:19.956718 10229 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:19.956768 10229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:19.956894 10229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:19.958256 10229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "7040f5a312a44afb81b3e56a87cea27e"
format_stamp: "Formatted at 2026-09-26 05:55:19 on dist-test-slave-7fr8"
server_key: "035bb3b65b34f24f72957461012de195"
server_key_iv: "34c20b3d1a9308e239757e1530ae8c62"
server_key_version: "encryptionkey@0"
I20260926 05:55:19.958647 10229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "7040f5a312a44afb81b3e56a87cea27e"
format_stamp: "Formatted at 2026-09-26 05:55:19 on dist-test-slave-7fr8"
server_key: "035bb3b65b34f24f72957461012de195"
server_key_iv: "34c20b3d1a9308e239757e1530ae8c62"
server_key_version: "encryptionkey@0"
I20260926 05:55:19.960145 10229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:55:19.960958 10244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:19.961161 10229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:19.961256 10229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "7040f5a312a44afb81b3e56a87cea27e"
format_stamp: "Formatted at 2026-09-26 05:55:19 on dist-test-slave-7fr8"
server_key: "035bb3b65b34f24f72957461012de195"
server_key_iv: "34c20b3d1a9308e239757e1530ae8c62"
server_key_version: "encryptionkey@0"
I20260926 05:55:19.961329 10229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:19.990031 10229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:19.992439 10229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:19.992604 10229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:19.992846 10229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:19.993289 10229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:19.993342 10229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:19.993376 10229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:19.993393 10229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:19.999174 10229 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:41105
I20260926 05:55:19.999221 10357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:41105 every 8 connection(s)
I20260926 05:55:19.999538 10229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:55:19.999763 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 10229
I20260926 05:55:19.999886 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:55:20.000121 25596 external_mini_cluster.cc:1428] Setting key 2971999c711ed86558bf5e4b2b07cbbf
Sep 26 05:55:20 dist-test-slave-7fr8 krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402119, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
I20260926 05:55:20.010800 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.000502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:33107 (local address 127.24.255.62:35731)
0926 05:55:20.000574 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:20.000577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:20.000756 (+ 179us) server_negotiation.cc:407] Connection header received
0926 05:55:20.001319 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:20.001321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:20.001337 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:20.001499 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:20.003059 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.003626 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.004701 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.004960 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.005042 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.007543 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:20.007546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:20.007546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:20.007558 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:20.009181 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:20.009618 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:20.009620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:20.009620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:20.009684 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:20.009984 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:20.009986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:20.009986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:20.010186 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:20.010254 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:20.010607 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 05:55:20.010723 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:55:20.011000 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.000441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35731 (local address 127.24.255.1:33107)
0926 05:55:20.000616 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:20.000667 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:20.001210 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:20.001701 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:20.001706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:20.002564 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:20.002900 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.002906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.003753 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.003754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.004591 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.004595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.005057 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.005058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.005199 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.005794 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:20.005804 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:20.007432 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:20.009274 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:20.009285 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:20.009288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:20.009539 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:20.009756 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:20.009758 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:20.009758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:20.009895 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:20.010283 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:20.010290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:20.010445 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 05:55:20.010570 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 05:55:20.010752 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260926 05:55:20.011489 10358 heartbeater.cc:347] Connected to a master server at 127.24.255.62:35731
I20260926 05:55:20.011636 10358 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
I20260926 05:55:20.011840 10358 heartbeater.cc:511] Master 127.24.255.62:35731 requested a full tablet report, sending...
I20260926 05:55:20.012465 10167 ts_manager.cc:194] Registered new tserver with Master: 7040f5a312a44afb81b3e56a87cea27e (127.24.255.1:41105)
I20260926 05:55:20.013303 10167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:33107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:55:20.029618 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35731
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:45255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:20.113325 10363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:20.113554 10363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:20.113577 10363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:20.114879 10363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:20.114930 10363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:20.114959 10363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:55:20.116460 10363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:45255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:20.116683 10363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:20.116869 10363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:55:20.122232 10363 server_base.cc:1034] running on GCE node
W20260926 05:55:20.122257 10371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:20.122262 10368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:20.122278 10369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:20.122682 10363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:20.122937 10363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:20.124123 10363 hybrid_clock.cc:648] HybridClock initialized: now 1790402120124069 us; error 71 us; skew 500 ppm
Sep 26 05:55:20 dist-test-slave-7fr8 krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402120, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:20.126772 10363 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:55:20.127312 10363 webserver.cc:460] Webserver started at http://127.24.255.2:38101/ using document root <none> and password file <none>
I20260926 05:55:20.127545 10363 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:20.127604 10363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:20.127732 10363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:20.128815 10363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "1524c084205d47aeab3bfa603b545505"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "420516261a4cff2d29c50f52b4a3792d"
server_key_iv: "4e3f779befbfdf638a5b5a785ce535ae"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.129170 10363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "1524c084205d47aeab3bfa603b545505"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "420516261a4cff2d29c50f52b4a3792d"
server_key_iv: "4e3f779befbfdf638a5b5a785ce535ae"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.130537 10363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:55:20.131410 10378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:20.131654 10363 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:20.131747 10363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "1524c084205d47aeab3bfa603b545505"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "420516261a4cff2d29c50f52b4a3792d"
server_key_iv: "4e3f779befbfdf638a5b5a785ce535ae"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.131820 10363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:20.138779 10363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:20.141112 10363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:20.141269 10363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:20.141500 10363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:20.141808 10363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:20.141841 10363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:20.141870 10363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:20.141894 10363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:20.147914 10363 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:46771
I20260926 05:55:20.147965 10491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:46771 every 8 connection(s)
I20260926 05:55:20.148308 10363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:55:20 dist-test-slave-7fr8 krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402120, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:20.154839 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 10363
I20260926 05:55:20.154935 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:55:20.155112 25596 external_mini_cluster.cc:1428] Setting key 682f3c0c3066d50703ef25789e895307
I20260926 05:55:20.157773 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.149070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:50989 (local address 127.24.255.62:35731)
0926 05:55:20.149154 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:20.149156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:20.149321 (+ 165us) server_negotiation.cc:407] Connection header received
0926 05:55:20.149842 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:20.149844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:20.149857 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:20.149915 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:20.150943 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.151471 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.152447 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.152642 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.152697 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.154945 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:20.154947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:20.154947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:20.154955 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:20.156303 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:20.156691 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:20.156692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:20.156693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:20.156779 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:20.157083 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:20.157084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:20.157084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:20.157222 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:20.157287 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:20.157583 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 05:55:20.157679 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 05:55:20.157966 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.149006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35731 (local address 127.24.255.2:50989)
0926 05:55:20.149194 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:20.149238 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:20.149739 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:20.149991 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:20.149996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:20.150641 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:20.150847 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.150851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.151561 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.151562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.152363 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.152365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.152713 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.152714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.152840 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.153379 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:20.153387 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:20.154852 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:20.156381 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:20.156390 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:20.156392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:20.156622 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:20.156866 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:20.156868 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:20.156868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:20.157019 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:20.157296 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:20.157303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:20.157454 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 05:55:20.157546 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 05:55:20.157737 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260926 05:55:20.158473 10492 heartbeater.cc:347] Connected to a master server at 127.24.255.62:35731
I20260926 05:55:20.158607 10492 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:20.158795 10492 heartbeater.cc:511] Master 127.24.255.62:35731 requested a full tablet report, sending...
I20260926 05:55:20.159173 10167 ts_manager.cc:194] Registered new tserver with Master: 1524c084205d47aeab3bfa603b545505 (127.24.255.2:46771)
I20260926 05:55:20.159705 10167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:50989
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:55:20.182245 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35731
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:45255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:20.262692 10497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:20.262909 10497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:20.262935 10497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:20.264220 10497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:20.264276 10497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:20.264304 10497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:55:20.265930 10497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:45255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:35731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:20.266382 10497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:20.266695 10497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:20.272034 10505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:20.272054 10503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:20.272015 10502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:20.272065 10497 server_base.cc:1034] running on GCE node
I20260926 05:55:20.272496 10497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:20.272773 10497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:20.273978 10497 hybrid_clock.cc:648] HybridClock initialized: now 1790402120273968 us; error 46 us; skew 500 ppm
Sep 26 05:55:20 dist-test-slave-7fr8 krb5kdc[10122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402120, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:20.276733 10497 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:55:20.277189 10497 webserver.cc:460] Webserver started at http://127.24.255.3:41823/ using document root <none> and password file <none>
I20260926 05:55:20.277390 10497 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:20.277428 10497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:20.277534 10497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:20.278631 10497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "0375713d739e46f999c2b91762bee040"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "e71221ea489809a4c9315ecc3496299c"
server_key_iv: "dbe7b2121c2a7dd82cd39edf03de92ef"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.278932 10497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "0375713d739e46f999c2b91762bee040"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "e71221ea489809a4c9315ecc3496299c"
server_key_iv: "dbe7b2121c2a7dd82cd39edf03de92ef"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.280282 10497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:55:20.281193 10512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:20.281498 10497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:20.281603 10497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "0375713d739e46f999c2b91762bee040"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "e71221ea489809a4c9315ecc3496299c"
server_key_iv: "dbe7b2121c2a7dd82cd39edf03de92ef"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.281678 10497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:20.298907 10497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:20.301154 10497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:20.301275 10497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:20.301470 10497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:20.301770 10497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:20.301807 10497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:20.301829 10497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:20.301839 10497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:20.307493 10497 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:38093
I20260926 05:55:20.307583 10625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:38093 every 8 connection(s)
I20260926 05:55:20.307850 10497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:55:20 dist-test-slave-7fr8 krb5kdc[10122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402120, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:20.317395 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 10497
I20260926 05:55:20.317494 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.308580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:35201 (local address 127.24.255.62:35731)
0926 05:55:20.308647 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:20.308649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:20.308838 (+ 189us) server_negotiation.cc:407] Connection header received
0926 05:55:20.309364 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:20.309366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:20.309377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:20.309465 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:20.310526 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.311045 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.312121 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.312321 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.312386 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.314697 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:20.314699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:20.314700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:20.314707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:20.316118 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:20.316542 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:20.316544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:20.316544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:20.316600 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:20.316804 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:20.316804 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:20.316804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:20.316916 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:20.316963 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:20.317313 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 05:55:20.317402 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:55:20.317515 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:55:20.317643 10628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.308539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:35731 (local address 127.24.255.3:35201)
0926 05:55:20.308718 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:20.308758 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:20.309262 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:20.309547 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:20.309551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:20.310211 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:20.310421 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.310426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.311147 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.311148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.312011 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.312014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.312397 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.312398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.312501 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.313045 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:20.313054 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:20.314564 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:20.316209 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:20.316217 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:20.316219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:20.316466 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:20.316651 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:20.316652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:20.316652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:20.316759 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:20.316991 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:20.316997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:20.317160 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 05:55:20.317264 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 05:55:20.317404 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260926 05:55:20.317691 25596 external_mini_cluster.cc:1428] Setting key cd380bc062b2238ee31b74e61ebc03b6
I20260926 05:55:20.318148 10626 heartbeater.cc:347] Connected to a master server at 127.24.255.62:35731
I20260926 05:55:20.318257 10626 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:20.318466 10626 heartbeater.cc:511] Master 127.24.255.62:35731 requested a full tablet report, sending...
I20260926 05:55:20.318838 10166 ts_manager.cc:194] Registered new tserver with Master: 0375713d739e46f999c2b91762bee040 (127.24.255.3:38093)
I20260926 05:55:20.319208 10166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:35201
I20260926 05:55:20.330135 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:55:20.337421 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.331581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.24.255.62:35731)
0926 05:55:20.331642 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:20.331643 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:20.331674 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:55:20.331739 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:20.331739 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:20.331748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:20.331801 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:20.332395 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.332959 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.333710 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.333932 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.333986 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.335173 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:20.335175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:20.335175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:20.335181 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:20.336492 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:20.336809 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:20.336810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:20.336810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:20.336859 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:20.336994 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:20.336994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:20.336994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:20.337146 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:20.337219 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:20.337292 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 05:55:20.337352 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:55:20.339155 10166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:55:20.340754 10166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:55:20.348342 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.344245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.24.255.2:46771)
0926 05:55:20.344400 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:20.344404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:20.344413 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:55:20.344449 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:20.344451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:20.344466 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:20.344692 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:20.344761 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.345567 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.346960 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.347894 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.347989 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.348060 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:20.348199 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 05:55:20.348247 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260926 05:55:20.348340 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.343760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:46771 (local address 127.0.0.1:42700)
0926 05:55:20.344177 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:20.344185 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:20.344220 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:20.344562 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:20.344564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:20.344566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:20.344624 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.344626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.345689 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.345690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.346865 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.346867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.348001 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.348002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.348067 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.348082 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 05:55:20.348172 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 05:55:20.348241 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":32,"thread_start_us":97,"threads_started":1}
I20260926 05:55:20.348838 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.343734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:38093 (local address 127.0.0.1:43322)
0926 05:55:20.343948 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:20.343999 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:20.344075 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:20.344272 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:20.344275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:20.344277 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:20.344353 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.344355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.345638 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.345639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.347092 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.347095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.348159 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.348160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.348228 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.348232 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:20.348274 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 05:55:20.348307 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":116,"thread_start_us":54,"threads_started":1}
I20260926 05:55:20.349248 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.343914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.24.255.3:38093)
0926 05:55:20.344105 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:20.344110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:20.344120 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:55:20.344162 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:20.344164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:20.344181 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:20.344256 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:20.344430 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.345535 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.347247 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.348060 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.348143 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.348199 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:20.348390 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 05:55:20.349173 (+ 783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
I20260926 05:55:20.349269 10426 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9ac1f2958f4bbd849da7fa0d3d86ff (DEFAULT_TABLE table=test-table [id=148b316ec9f84f15859c7a70e7334f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:20.349440 10637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.343734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:41105 (local address 127.0.0.1:49230)
0926 05:55:20.343992 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:20.344002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:20.344086 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:20.344757 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:20.344758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:20.344759 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:20.344793 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.344795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.345987 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.345988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.347091 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.347095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.349172 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.349173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.349231 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.349235 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:20.349305 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 05:55:20.349354 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":2,"spinlock_wait_cycles":896,"thread_start_us":117,"threads_started":1}
I20260926 05:55:20.349459 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.344040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.24.255.1:41105)
0926 05:55:20.344540 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:20.344543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:20.344552 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:55:20.344586 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:20.344588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:20.344602 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:20.344679 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:20.344870 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.345891 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.348246 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.349000 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.349071 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.349134 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:20.349335 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 05:55:20.349381 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":83,"threads_started":1}
I20260926 05:55:20.349661 10426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9ac1f2958f4bbd849da7fa0d3d86ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:20.350139 10560 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9ac1f2958f4bbd849da7fa0d3d86ff (DEFAULT_TABLE table=test-table [id=148b316ec9f84f15859c7a70e7334f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:20.350126 10292 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9ac1f2958f4bbd849da7fa0d3d86ff (DEFAULT_TABLE table=test-table [id=148b316ec9f84f15859c7a70e7334f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:20.350486 10560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9ac1f2958f4bbd849da7fa0d3d86ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:20.350486 10292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9ac1f2958f4bbd849da7fa0d3d86ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:20.352401 10643 tablet_bootstrap.cc:492] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505: Bootstrap starting.
I20260926 05:55:20.353015 10643 tablet_bootstrap.cc:654] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:20.353472 10643 log.cc:824] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:20.353850 10645 tablet_bootstrap.cc:492] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040: Bootstrap starting.
I20260926 05:55:20.354252 10643 tablet_bootstrap.cc:492] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505: No bootstrap required, opened a new log
I20260926 05:55:20.354310 10643 ts_tablet_manager.cc:1397] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:55:20.354511 10645 tablet_bootstrap.cc:654] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:20.355072 10645 log.cc:824] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:20.355403 10646 tablet_bootstrap.cc:492] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e: Bootstrap starting.
I20260926 05:55:20.355784 10645 tablet_bootstrap.cc:492] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040: No bootstrap required, opened a new log
I20260926 05:55:20.355829 10645 ts_tablet_manager.cc:1397] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:55:20.356009 10646 tablet_bootstrap.cc:654] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:20.356357 10643 raft_consensus.cc:348] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5a312a44afb81b3e56a87cea27e" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41105 } } peers { permanent_uuid: "0375713d739e46f999c2b91762bee040" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38093 } } peers { permanent_uuid: "1524c084205d47aeab3bfa603b545505" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46771 } }
I20260926 05:55:20.356537 10643 raft_consensus.cc:374] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:20.356539 10646 log.cc:824] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:20.356575 10643 raft_consensus.cc:729] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1524c084205d47aeab3bfa603b545505, State: Initialized, Role: FOLLOWER
I20260926 05:55:20.356698 10643 consensus_queue.cc:260] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5a312a44afb81b3e56a87cea27e" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41105 } } peers { permanent_uuid: "0375713d739e46f999c2b91762bee040" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38093 } } peers { permanent_uuid: "1524c084205d47aeab3bfa603b545505" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46771 } }
I20260926 05:55:20.356967 10492 heartbeater.cc:503] Master 127.24.255.62:35731 was elected leader, sending a full tablet report...
I20260926 05:55:20.357074 10643 ts_tablet_manager.cc:1428] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:55:20.357471 10646 tablet_bootstrap.cc:492] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e: No bootstrap required, opened a new log
I20260926 05:55:20.357527 10646 ts_tablet_manager.cc:1397] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:55:20.357412 10645 raft_consensus.cc:348] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5a312a44afb81b3e56a87cea27e" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41105 } } peers { permanent_uuid: "0375713d739e46f999c2b91762bee040" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38093 } } peers { permanent_uuid: "1524c084205d47aeab3bfa603b545505" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46771 } }
I20260926 05:55:20.357589 10645 raft_consensus.cc:374] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:20.357633 10645 raft_consensus.cc:729] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0375713d739e46f999c2b91762bee040, State: Initialized, Role: FOLLOWER
I20260926 05:55:20.357761 10645 consensus_queue.cc:260] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5a312a44afb81b3e56a87cea27e" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41105 } } peers { permanent_uuid: "0375713d739e46f999c2b91762bee040" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38093 } } peers { permanent_uuid: "1524c084205d47aeab3bfa603b545505" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46771 } }
I20260926 05:55:20.358016 10645 ts_tablet_manager.cc:1428] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:55:20.358037 10626 heartbeater.cc:503] Master 127.24.255.62:35731 was elected leader, sending a full tablet report...
I20260926 05:55:20.359277 10646 raft_consensus.cc:348] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5a312a44afb81b3e56a87cea27e" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41105 } } peers { permanent_uuid: "0375713d739e46f999c2b91762bee040" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38093 } } peers { permanent_uuid: "1524c084205d47aeab3bfa603b545505" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46771 } }
I20260926 05:55:20.359436 10646 raft_consensus.cc:374] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:20.359469 10646 raft_consensus.cc:729] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7040f5a312a44afb81b3e56a87cea27e, State: Initialized, Role: FOLLOWER
I20260926 05:55:20.359575 10646 consensus_queue.cc:260] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5a312a44afb81b3e56a87cea27e" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41105 } } peers { permanent_uuid: "0375713d739e46f999c2b91762bee040" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38093 } } peers { permanent_uuid: "1524c084205d47aeab3bfa603b545505" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46771 } }
I20260926 05:55:20.359810 10646 ts_tablet_manager.cc:1428] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:55:20.359858 10358 heartbeater.cc:503] Master 127.24.255.62:35731 was elected leader, sending a full tablet report...
W20260926 05:55:20.398907 10493 tablet.cc:2369] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:55:20.500392 10359 tablet.cc:2369] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:55:20.520421 10649 raft_consensus.cc:482] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:55:20.520570 10649 raft_consensus.cc:504] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5a312a44afb81b3e56a87cea27e" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41105 } } peers { permanent_uuid: "0375713d739e46f999c2b91762bee040" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38093 } } peers { permanent_uuid: "1524c084205d47aeab3bfa603b545505" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46771 } }
I20260926 05:55:20.521009 10649 leader_election.cc:288] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7040f5a312a44afb81b3e56a87cea27e (127.24.255.1:41105), 0375713d739e46f999c2b91762bee040 (127.24.255.3:38093)
I20260926 05:55:20.523999 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.521226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:41105 (local address 127.24.255.2:53531)
0926 05:55:20.521438 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:20.521448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:20.521523 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:20.521653 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:20.521654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:20.521655 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:20.521727 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.521728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.522451 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.522452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.523228 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.523229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.523756 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.523757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.523826 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.523831 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:55:20.523882 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:55:20.523924 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":17,"thread_start_us":123,"threads_started":1}
I20260926 05:55:20.524067 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.521354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:53531 (local address 127.24.255.1:41105)
0926 05:55:20.521445 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:20.521448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:20.521528 (+ 80us) server_negotiation.cc:407] Connection header received
0926 05:55:20.521582 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:20.521583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:20.521593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:20.521641 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:20.521823 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.522351 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.523319 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.523677 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.523729 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.523737 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:20.523920 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 05:55:20.523995 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 05:55:20.524173 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.521226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:38093 (local address 127.24.255.2:35833)
0926 05:55:20.521306 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:20.521319 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:20.521360 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:20.521542 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:20.521543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:20.521545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:20.521625 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.521628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.522351 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.522352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.523241 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.523244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.523900 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.523901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.523999 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.524003 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:20.524068 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 05:55:20.524111 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":4}
I20260926 05:55:20.524322 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.521351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:35833 (local address 127.24.255.3:38093)
0926 05:55:20.521398 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:20.521400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:20.521411 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:55:20.521459 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:20.521460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:20.521470 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:20.521574 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:20.521688 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.522265 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.523346 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.523797 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.523871 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.523879 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:20.524212 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 05:55:20.524257 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:55:20.524552 10312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9ac1f2958f4bbd849da7fa0d3d86ff" candidate_uuid: "1524c084205d47aeab3bfa603b545505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7040f5a312a44afb81b3e56a87cea27e" is_pre_election: true
I20260926 05:55:20.524693 10312 raft_consensus.cc:2454] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1524c084205d47aeab3bfa603b545505 in term 0.
I20260926 05:55:20.524883 10580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9ac1f2958f4bbd849da7fa0d3d86ff" candidate_uuid: "1524c084205d47aeab3bfa603b545505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0375713d739e46f999c2b91762bee040" is_pre_election: true
I20260926 05:55:20.524917 10379 leader_election.cc:302] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1524c084205d47aeab3bfa603b545505, 7040f5a312a44afb81b3e56a87cea27e; no voters:
I20260926 05:55:20.524995 10580 raft_consensus.cc:2454] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1524c084205d47aeab3bfa603b545505 in term 0.
I20260926 05:55:20.525056 10649 raft_consensus.cc:2781] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:55:20.525146 10649 raft_consensus.cc:482] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:55:20.525174 10649 raft_consensus.cc:3037] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:20.525870 10649 raft_consensus.cc:504] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5a312a44afb81b3e56a87cea27e" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41105 } } peers { permanent_uuid: "0375713d739e46f999c2b91762bee040" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38093 } } peers { permanent_uuid: "1524c084205d47aeab3bfa603b545505" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46771 } }
I20260926 05:55:20.525993 10649 leader_election.cc:288] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [CANDIDATE]: Term 1 election: Requested vote from peers 7040f5a312a44afb81b3e56a87cea27e (127.24.255.1:41105), 0375713d739e46f999c2b91762bee040 (127.24.255.3:38093)
I20260926 05:55:20.526228 10580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9ac1f2958f4bbd849da7fa0d3d86ff" candidate_uuid: "1524c084205d47aeab3bfa603b545505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0375713d739e46f999c2b91762bee040"
I20260926 05:55:20.526233 10312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9ac1f2958f4bbd849da7fa0d3d86ff" candidate_uuid: "1524c084205d47aeab3bfa603b545505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7040f5a312a44afb81b3e56a87cea27e"
I20260926 05:55:20.526305 10580 raft_consensus.cc:3037] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:20.526319 10312 raft_consensus.cc:3037] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:20.527045 10580 raft_consensus.cc:2454] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1524c084205d47aeab3bfa603b545505 in term 1.
I20260926 05:55:20.527053 10312 raft_consensus.cc:2454] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1524c084205d47aeab3bfa603b545505 in term 1.
I20260926 05:55:20.527210 10379 leader_election.cc:302] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1524c084205d47aeab3bfa603b545505, 7040f5a312a44afb81b3e56a87cea27e; no voters:
I20260926 05:55:20.527313 10649 raft_consensus.cc:2781] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:20.527475 10649 raft_consensus.cc:686] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [term 1 LEADER]: Becoming Leader. State: Replica: 1524c084205d47aeab3bfa603b545505, State: Running, Role: LEADER
I20260926 05:55:20.527554 10649 consensus_queue.cc:237] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5a312a44afb81b3e56a87cea27e" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41105 } } peers { permanent_uuid: "0375713d739e46f999c2b91762bee040" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38093 } } peers { permanent_uuid: "1524c084205d47aeab3bfa603b545505" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46771 } }
I20260926 05:55:20.528401 10166 catalog_manager.cc:5495] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1524c084205d47aeab3bfa603b545505 (127.24.255.2). New cstate: current_term: 1 leader_uuid: "1524c084205d47aeab3bfa603b545505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5a312a44afb81b3e56a87cea27e" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0375713d739e46f999c2b91762bee040" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1524c084205d47aeab3bfa603b545505" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46771 } health_report { overall_health: HEALTHY } } }
I20260926 05:55:20.533622 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.531287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.24.255.2:46771)
0926 05:55:20.531336 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:20.531338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:20.531371 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:55:20.531400 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:20.531400 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:20.531416 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:20.531466 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:20.531570 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.532112 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.532909 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.533106 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.533163 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.533320 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:20.533429 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:20.533500 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:20.533521 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 05:55:20.533541 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:55:20.535437 10580 raft_consensus.cc:1261] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040 [term 1 FOLLOWER]: Refusing update from remote peer 1524c084205d47aeab3bfa603b545505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:20.535454 10312 raft_consensus.cc:1261] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e [term 1 FOLLOWER]: Refusing update from remote peer 1524c084205d47aeab3bfa603b545505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:20.535845 10653 consensus_queue.cc:1035] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0375713d739e46f999c2b91762bee040" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:20.535979 10649 consensus_queue.cc:1035] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7040f5a312a44afb81b3e56a87cea27e" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:20.538743 10656 mvcc.cc:205] Tried to move back new op lower bound from 7333487085710491648 to 7333487085681197056. Current Snapshot: MvccSnapshot[applied={T|T < 7333487085710491648}]
I20260926 05:55:20.538798 10657 mvcc.cc:205] Tried to move back new op lower bound from 7333487085710491648 to 7333487085681197056. Current Snapshot: MvccSnapshot[applied={T|T < 7333487085710491648}]
I20260926 05:55:20.538890 10658 mvcc.cc:205] Tried to move back new op lower bound from 7333487085710491648 to 7333487085681197056. Current Snapshot: MvccSnapshot[applied={T|T < 7333487085710491648 or (T in {7333487085710491648})}]
I20260926 05:55:20.541406 10167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:55:20.541519 10167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:55:20.542377 10167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a152d1af0b2b405f9196e9e7c865b3d3: Sending DeleteTablet for 3 replicas of tablet 9e9ac1f2958f4bbd849da7fa0d3d86ff
I20260926 05:55:20.542837 10292 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e9ac1f2958f4bbd849da7fa0d3d86ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:20 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:49230
I20260926 05:55:20.542837 10560 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e9ac1f2958f4bbd849da7fa0d3d86ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:20 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:43322
I20260926 05:55:20.542958 10426 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e9ac1f2958f4bbd849da7fa0d3d86ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:20 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:42700
I20260926 05:55:20.543212 10666 tablet_replica.cc:330] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 7040f5a312a44afb81b3e56a87cea27e: stopping tablet replica
I20260926 05:55:20.543272 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 10229
I20260926 05:55:20.544171 10668 tablet_replica.cc:330] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505: stopping tablet replica
I20260926 05:55:20.544272 10668 raft_consensus.cc:2229] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:55:20.544395 10668 raft_consensus.cc:2258] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:20.544617 10667 tablet_replica.cc:330] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040: stopping tablet replica
I20260926 05:55:20.544759 10667 raft_consensus.cc:2229] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:55:20.544842 10668 ts_tablet_manager.cc:1905] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:55:20.544867 10667 raft_consensus.cc:2258] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:20.545322 10667 ts_tablet_manager.cc:1905] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:55:20.546581 10668 ts_tablet_manager.cc:1918] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:55:20.546657 10668 log.cc:1196] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/9e9ac1f2958f4bbd849da7fa0d3d86ff
I20260926 05:55:20.546906 10668 ts_tablet_manager.cc:1939] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 1524c084205d47aeab3bfa603b545505: Deleting consensus metadata
I20260926 05:55:20.547104 10667 ts_tablet_manager.cc:1918] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:55:20.547178 10667 log.cc:1196] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/9e9ac1f2958f4bbd849da7fa0d3d86ff
I20260926 05:55:20.547307 10152 catalog_manager.cc:4841] TS 1524c084205d47aeab3bfa603b545505 (127.24.255.2:46771): tablet 9e9ac1f2958f4bbd849da7fa0d3d86ff (table test-table [id=148b316ec9f84f15859c7a70e7334f97]) successfully deleted
I20260926 05:55:20.547441 10667 ts_tablet_manager.cc:1939] T 9e9ac1f2958f4bbd849da7fa0d3d86ff P 0375713d739e46f999c2b91762bee040: Deleting consensus metadata
I20260926 05:55:20.547973 10153 catalog_manager.cc:4841] TS 0375713d739e46f999c2b91762bee040 (127.24.255.3:38093): tablet 9e9ac1f2958f4bbd849da7fa0d3d86ff (table test-table [id=148b316ec9f84f15859c7a70e7334f97]) successfully deleted
W20260926 05:55:20.548808 10151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:41105 (error 108)
I20260926 05:55:20.549322 10637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.549169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:41105 (local address 127.0.0.1:49234)
0926 05:55:20.549243 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:20.549291 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:41105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 05:55:20.549461 10151 catalog_manager.cc:4584] TS 7040f5a312a44afb81b3e56a87cea27e (127.24.255.1:41105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e9ac1f2958f4bbd849da7fa0d3d86ff: Network error: Client connection negotiation failed: client connection to 127.24.255.1:41105: connect: Connection refused (error 111)
I20260926 05:55:20.549966 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 10363
I20260926 05:55:20.554628 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 10497
I20260926 05:55:20.559379 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 10135
2026-09-26T05:55:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2011 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T05:55:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:20Z Disabled control of system clock
I20260926 05:55:20.585825 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:38979
--webserver_interface=127.24.255.62
--webserver_port=0
--builtin_ntp_servers=127.24.255.20:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:38979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:55:20.671226 10673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:20.671470 10673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:20.671492 10673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:20.672693 10673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:20.672737 10673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:20.672749 10673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:20.672760 10673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:20.672771 10673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:20.674278 10673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:38979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:38979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:20.674525 10673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:20.674798 10673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:20.679766 10681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:20.679832 10673 server_base.cc:1034] running on GCE node
W20260926 05:55:20.679759 10678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:20.679759 10679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:20.680266 10673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:20.680578 10673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:20.681761 10673 hybrid_clock.cc:648] HybridClock initialized: now 1790402120681743 us; error 34 us; skew 500 ppm
I20260926 05:55:20.683823 10673 webserver.cc:460] Webserver started at http://127.24.255.62:35991/ using document root <none> and password file <none>
I20260926 05:55:20.684093 10673 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:20.684149 10673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:20.684298 10673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:20.685418 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "c1f25b9fc171414bbf4a3a48e5d707cd"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "094c08435a0b8c977dabde333b8925d3"
server_key_iv: "a8671dabdcac34fe9aa3f29fa33931f3"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.685776 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "c1f25b9fc171414bbf4a3a48e5d707cd"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "094c08435a0b8c977dabde333b8925d3"
server_key_iv: "a8671dabdcac34fe9aa3f29fa33931f3"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.687218 10673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:55:20.688083 10687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:20.688320 10673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:55:20.688406 10673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "c1f25b9fc171414bbf4a3a48e5d707cd"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "094c08435a0b8c977dabde333b8925d3"
server_key_iv: "a8671dabdcac34fe9aa3f29fa33931f3"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.688475 10673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:20.695292 10673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:20.695605 10673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:20.695732 10673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:20.699622 10673 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:38979
I20260926 05:55:20.699677 10739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:38979 every 8 connection(s)
I20260926 05:55:20.700151 10673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:20.700492 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 10673
I20260926 05:55:20.700601 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:20.700759 25596 external_mini_cluster.cc:1428] Setting key 236622697021a6bd5781f41911a30ff9
I20260926 05:55:20.701689 10740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:20.705511 10740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd: Bootstrap starting.
I20260926 05:55:20.706413 10740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:20.706746 10740 log.cc:824] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:20.707453 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.701444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51486 (local address 127.24.255.62:38979)
0926 05:55:20.701730 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:20.701739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:20.701774 (+ 35us) server_negotiation.cc:407] Connection header received
0926 05:55:20.702401 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:20.702411 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:20.702497 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:20.702674 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:20.702869 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.704205 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.705128 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.705828 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.705970 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.706720 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:20.706725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:20.706726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:20.706949 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:20.706993 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:20.707010 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:20.707070 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:20.707165 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 05:55:20.707236 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":110,"threads_started":1}
I20260926 05:55:20.707540 10740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd: No bootstrap required, opened a new log
I20260926 05:55:20.709513 10740 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f25b9fc171414bbf4a3a48e5d707cd" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38979 } }
I20260926 05:55:20.709681 10740 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:20.709704 10740 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f25b9fc171414bbf4a3a48e5d707cd, State: Initialized, Role: FOLLOWER
I20260926 05:55:20.709802 10740 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f25b9fc171414bbf4a3a48e5d707cd" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38979 } }
I20260926 05:55:20.709872 10740 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:20.709894 10740 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:20.709928 10740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:20.711038 10740 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f25b9fc171414bbf4a3a48e5d707cd" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38979 } }
I20260926 05:55:20.711398 10740 leader_election.cc:302] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1f25b9fc171414bbf4a3a48e5d707cd; no voters:
I20260926 05:55:20.711651 10740 leader_election.cc:288] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:20.711709 10745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:20.711881 10745 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [term 1 LEADER]: Becoming Leader. State: Replica: c1f25b9fc171414bbf4a3a48e5d707cd, State: Running, Role: LEADER
I20260926 05:55:20.711961 10740 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:20.711982 10745 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f25b9fc171414bbf4a3a48e5d707cd" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38979 } }
I20260926 05:55:20.712361 10746 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1f25b9fc171414bbf4a3a48e5d707cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f25b9fc171414bbf4a3a48e5d707cd" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38979 } } }
I20260926 05:55:20.712543 10747 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1f25b9fc171414bbf4a3a48e5d707cd. Latest consensus state: current_term: 1 leader_uuid: "c1f25b9fc171414bbf4a3a48e5d707cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f25b9fc171414bbf4a3a48e5d707cd" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38979 } } }
I20260926 05:55:20.712646 10747 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:20.712543 10746 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:20.713018 10754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:20.713538 10754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:20.715493 10754 catalog_manager.cc:1347] Generated new cluster ID: b176cd0d6aa74b21a4cba4d031d02d56
I20260926 05:55:20.715576 10754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:20.733776 10754 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:20.734545 10754 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:20.754411 10754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd: Generated new TSK 0
I20260926 05:55:20.754751 10754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:20.758960 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38979
--builtin_ntp_servers=127.24.255.20:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:55:20.841990 10764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:20.842286 10764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:20.842312 10764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:20.843580 10764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:20.843636 10764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:20.843667 10764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:55:20.845131 10764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:20.845396 10764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:20.845746 10764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:20.850940 10769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:20.850935 10770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:20.850932 10772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:20.851086 10764 server_base.cc:1034] running on GCE node
I20260926 05:55:20.851306 10764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:20.851567 10764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:20.852725 10764 hybrid_clock.cc:648] HybridClock initialized: now 1790402120852709 us; error 32 us; skew 500 ppm
I20260926 05:55:20.854010 10764 webserver.cc:460] Webserver started at http://127.24.255.1:40403/ using document root <none> and password file <none>
I20260926 05:55:20.854288 10764 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:20.854341 10764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:20.854508 10764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:20.855572 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "4700bea51b59462da74eb364c878b71d"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "c680edf98e6f43f92382046431b4cc1a"
server_key_iv: "ced143902911001376cbb4833846ce40"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.855894 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "4700bea51b59462da74eb364c878b71d"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "c680edf98e6f43f92382046431b4cc1a"
server_key_iv: "ced143902911001376cbb4833846ce40"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.857168 10764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:55:20.857990 10778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:20.858192 10764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:20.858286 10764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "4700bea51b59462da74eb364c878b71d"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "c680edf98e6f43f92382046431b4cc1a"
server_key_iv: "ced143902911001376cbb4833846ce40"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.858357 10764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:20.873131 10764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:20.873485 10764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:20.873627 10764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:20.873852 10764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:20.874238 10764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:20.874275 10764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:20.874306 10764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:20.874330 10764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:20.880089 10764 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:38315
I20260926 05:55:20.880208 10891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:38315 every 8 connection(s)
I20260926 05:55:20.880481 10764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:55:20.884346 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 10764
I20260926 05:55:20.884456 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:55:20.884641 25596 external_mini_cluster.cc:1428] Setting key ecaac7d3a44569d309a82e4e1b9ee630
I20260926 05:55:20.885777 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38979
--builtin_ntp_servers=127.24.255.20:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 05:55:20.886768 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.881248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:33457 (local address 127.24.255.62:38979)
0926 05:55:20.881367 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:20.881369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:20.881520 (+ 151us) server_negotiation.cc:407] Connection header received
0926 05:55:20.881995 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:20.881996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:20.882012 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:20.882085 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:20.883030 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.883598 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.884990 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.885235 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.885302 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.886452 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:20.886453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:20.886454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:20.886458 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:20.886473 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:20.886477 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:20.886530 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:20.886664 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 05:55:20.886704 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 05:55:20.887002 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:20.881189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38979 (local address 127.24.255.1:33457)
0926 05:55:20.881376 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:20.881419 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:20.881894 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:20.882224 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:20.882229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:20.882238 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:20.882926 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.882932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.883707 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.883708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.884866 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:20.884869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:20.885384 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:20.885385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:20.885594 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:20.886266 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:55:20.886267 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:20.886311 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:55:20.886344 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:20.886540 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:20.886545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:20.886548 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:55:20.886612 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:55:20.886731 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260926 05:55:20.887657 10892 heartbeater.cc:347] Connected to a master server at 127.24.255.62:38979
I20260926 05:55:20.887756 10892 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:20.888029 10892 heartbeater.cc:511] Master 127.24.255.62:38979 requested a full tablet report, sending...
I20260926 05:55:20.888608 10704 ts_manager.cc:194] Registered new tserver with Master: 4700bea51b59462da74eb364c878b71d (127.24.255.1:38315)
I20260926 05:55:20.889240 10704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.255.1:33457
W20260926 05:55:20.973719 10895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:20.973951 10895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:20.973984 10895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:20.975328 10895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:20.975389 10895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:20.975442 10895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:55:20.976948 10895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:20.977327 10895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:20.977746 10895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:20.983287 10901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:20.983428 10903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:20.983554 10895 server_base.cc:1034] running on GCE node
W20260926 05:55:20.983299 10900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:20.983968 10895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:20.984383 10895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:20.985590 10895 hybrid_clock.cc:648] HybridClock initialized: now 1790402120985573 us; error 40 us; skew 500 ppm
I20260926 05:55:20.986949 10895 webserver.cc:460] Webserver started at http://127.24.255.2:44381/ using document root <none> and password file <none>
I20260926 05:55:20.987210 10895 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:20.987262 10895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:20.987370 10895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:20.988468 10895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "b278530dac2048ca885ecf11f6255909"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "1829f10dfd47cc4f1f78d620c1810c08"
server_key_iv: "6474183f8c498ff3a36fbd69c651e91c"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.988792 10895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "b278530dac2048ca885ecf11f6255909"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "1829f10dfd47cc4f1f78d620c1810c08"
server_key_iv: "6474183f8c498ff3a36fbd69c651e91c"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.990132 10895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:55:20.990936 10909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:20.991250 10895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:55:20.991340 10895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "b278530dac2048ca885ecf11f6255909"
format_stamp: "Formatted at 2026-09-26 05:55:20 on dist-test-slave-7fr8"
server_key: "1829f10dfd47cc4f1f78d620c1810c08"
server_key_iv: "6474183f8c498ff3a36fbd69c651e91c"
server_key_version: "encryptionkey@0"
I20260926 05:55:20.991400 10895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:21.008177 10895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:21.008528 10895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:21.008677 10895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:21.008929 10895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:21.009271 10895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:21.009308 10895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:21.009342 10895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:21.009370 10895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:21.015796 10895 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:37949
I20260926 05:55:21.015883 11022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:37949 every 8 connection(s)
I20260926 05:55:21.016332 10895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:55:21.021901 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 10895
I20260926 05:55:21.022027 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:55:21.022257 25596 external_mini_cluster.cc:1428] Setting key 3203db27d76de6653552fc0aebab2622
I20260926 05:55:21.023150 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.017322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:59395 (local address 127.24.255.62:38979)
0926 05:55:21.017391 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:21.017393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:21.017762 (+ 369us) server_negotiation.cc:407] Connection header received
0926 05:55:21.018451 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:21.018452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:21.018465 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:21.018533 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:21.019664 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.020199 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.021360 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.021612 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.021677 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.022839 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:21.022841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:21.022842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:21.022847 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:21.022865 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:21.022869 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:21.022921 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:21.023034 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 05:55:21.023067 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:55:21.023239 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38979
--builtin_ntp_servers=127.24.255.20:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 05:55:21.023389 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.017311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38979 (local address 127.24.255.2:59395)
0926 05:55:21.017614 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:21.017654 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:21.018263 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:21.018707 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:21.018713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:21.018727 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:21.019530 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.019540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.020317 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.020319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.021248 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.021251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.021736 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.021738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.021920 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.022691 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:55:21.022692 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:21.022726 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:55:21.022754 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:21.022944 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:21.022950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:21.022953 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:55:21.023024 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 05:55:21.023149 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":161,"threads_started":1}
I20260926 05:55:21.023916 11023 heartbeater.cc:347] Connected to a master server at 127.24.255.62:38979
I20260926 05:55:21.024026 11023 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:21.024217 11023 heartbeater.cc:511] Master 127.24.255.62:38979 requested a full tablet report, sending...
I20260926 05:55:21.024688 10704 ts_manager.cc:194] Registered new tserver with Master: b278530dac2048ca885ecf11f6255909 (127.24.255.2:37949)
I20260926 05:55:21.025075 10704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.255.2:59395
W20260926 05:55:21.107172 11026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:21.107431 11026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:21.107467 11026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:21.108757 11026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:21.108815 11026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:21.108856 11026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:55:21.110976 11026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:21.111279 11026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:21.111778 11026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:21.117262 11034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:21.117398 11032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:21.117398 11031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:21.117285 11026 server_base.cc:1034] running on GCE node
I20260926 05:55:21.117794 11026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:21.118364 11026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:21.119632 11026 hybrid_clock.cc:648] HybridClock initialized: now 1790402121119616 us; error 28 us; skew 500 ppm
I20260926 05:55:21.121143 11026 webserver.cc:460] Webserver started at http://127.24.255.3:45731/ using document root <none> and password file <none>
I20260926 05:55:21.121394 11026 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:21.121446 11026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:21.121583 11026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:21.122856 11026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "6ed2de89e81f4149a65f7b341a7e8831"
format_stamp: "Formatted at 2026-09-26 05:55:21 on dist-test-slave-7fr8"
server_key: "9e71ca4b0748917e3ade8943b491e86a"
server_key_iv: "b5032b46ac9b40038569177eeac09041"
server_key_version: "encryptionkey@0"
I20260926 05:55:21.123221 11026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "6ed2de89e81f4149a65f7b341a7e8831"
format_stamp: "Formatted at 2026-09-26 05:55:21 on dist-test-slave-7fr8"
server_key: "9e71ca4b0748917e3ade8943b491e86a"
server_key_iv: "b5032b46ac9b40038569177eeac09041"
server_key_version: "encryptionkey@0"
I20260926 05:55:21.124819 11026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:55:21.125820 11040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:21.126248 11026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:21.126353 11026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "6ed2de89e81f4149a65f7b341a7e8831"
format_stamp: "Formatted at 2026-09-26 05:55:21 on dist-test-slave-7fr8"
server_key: "9e71ca4b0748917e3ade8943b491e86a"
server_key_iv: "b5032b46ac9b40038569177eeac09041"
server_key_version: "encryptionkey@0"
I20260926 05:55:21.126453 11026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:21.160137 11026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:21.160486 11026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:21.160626 11026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:21.160861 11026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:21.161198 11026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:21.161237 11026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:21.161269 11026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:21.161289 11026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:21.167451 11026 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:38173
I20260926 05:55:21.167502 11153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:38173 every 8 connection(s)
I20260926 05:55:21.167822 11026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:55:21.169142 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 11026
I20260926 05:55:21.169231 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:55:21.169392 25596 external_mini_cluster.cc:1428] Setting key b45be0612d62bb5410f4a3699ebbc240
I20260926 05:55:21.173369 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.168640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:35065 (local address 127.24.255.62:38979)
0926 05:55:21.168720 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:21.168722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:21.169004 (+ 282us) server_negotiation.cc:407] Connection header received
0926 05:55:21.169526 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:21.169527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:21.169539 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:21.169585 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:21.170466 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.170979 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.171976 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.172176 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.172239 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.173089 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:21.173091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:21.173091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:21.173097 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:21.173114 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:21.173118 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:21.173167 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:21.173249 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 05:55:21.173279 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 05:55:21.173658 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.168640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38979 (local address 127.24.255.3:35065)
0926 05:55:21.168818 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:21.168861 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:21.169418 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:21.169656 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:21.169660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:21.169670 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:21.170358 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.170365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.171072 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.171073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.171884 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.171887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.172251 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.172253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.172401 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.172940 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:55:21.172940 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:21.172969 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:55:21.172994 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:21.173173 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:21.173178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:21.173182 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:21.173248 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 05:55:21.173362 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260926 05:55:21.174202 11154 heartbeater.cc:347] Connected to a master server at 127.24.255.62:38979
I20260926 05:55:21.174300 11154 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:21.174507 11154 heartbeater.cc:511] Master 127.24.255.62:38979 requested a full tablet report, sending...
I20260926 05:55:21.175026 10704 ts_manager.cc:194] Registered new tserver with Master: 6ed2de89e81f4149a65f7b341a7e8831 (127.24.255.3:38173)
I20260926 05:55:21.175369 10704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.255.3:35065
I20260926 05:55:21.181799 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:55:21.186439 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.183367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.24.255.62:38979)
0926 05:55:21.183464 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:21.183467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:21.183548 (+ 81us) server_negotiation.cc:407] Connection header received
0926 05:55:21.183618 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:21.183619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:21.183631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:21.183694 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:21.183870 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.184400 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.185198 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.185406 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.185469 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.186197 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:21.186199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:21.186199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:21.186205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:21.186220 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:21.186224 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:21.186267 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:21.186326 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 05:55:21.186351 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 05:55:21.188730 10704 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:55:21.190402 10704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:55:21.197356 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.193000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.24.255.2:37949)
0926 05:55:21.193318 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:21.193324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:21.193387 (+ 63us) server_negotiation.cc:407] Connection header received
0926 05:55:21.193779 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:21.193781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:21.193798 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:21.193873 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:21.193976 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.195096 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.196193 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.196793 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.196858 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.196956 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:21.197047 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 05:55:21.197087 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":128,"threads_started":1}
I20260926 05:55:21.197170 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.192890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:37949 (local address 127.0.0.1:41098)
0926 05:55:21.193327 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:21.193337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:21.193398 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:21.193881 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:21.193882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:21.193884 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:21.193918 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.193919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.195185 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.195186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.196121 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.196124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.196865 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.196866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.196951 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.196966 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 05:55:21.197034 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 05:55:21.197089 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":23,"thread_start_us":342,"threads_started":1}
I20260926 05:55:21.197535 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.193159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.24.255.1:38315)
0926 05:55:21.193431 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:21.193436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:21.193443 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:55:21.193473 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:21.193475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:21.193488 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:21.193549 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:21.193732 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.194662 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.196107 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.196794 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.196990 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.197115 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:21.197379 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 05:55:21.197426 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":76,"threads_started":1}
I20260926 05:55:21.197641 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.192890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:38315 (local address 127.0.0.1:43398)
0926 05:55:21.193276 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:21.193284 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:21.193324 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:21.193576 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:21.193579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:21.193581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:21.193653 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.193655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.194739 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.194740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.196015 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.196018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.197161 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.197162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.197227 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.197232 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:55:21.197290 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 05:55:21.197335 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":88,"threads_started":1}
I20260926 05:55:21.198191 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.192885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:38173 (local address 127.0.0.1:54792)
0926 05:55:21.193054 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:21.193094 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:21.193162 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:21.193662 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:21.193662 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:21.193664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:21.193701 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.193702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.195176 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.195177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.196307 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.196309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.197345 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.197345 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.197398 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.197402 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:21.198044 (+ 642us) client_negotiation.cc:240] Negotiation successful
0926 05:55:21.198119 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":34,"thread_start_us":104,"threads_started":1}
I20260926 05:55:21.198187 10957 tablet_service.cc:1467] Processing CreateTablet for tablet edbdc723a0044e3998928dea71fe03b8 (DEFAULT_TABLE table=test-table [id=ae45bbb805db4004af6ca6297994075a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:21.198298 10826 tablet_service.cc:1467] Processing CreateTablet for tablet edbdc723a0044e3998928dea71fe03b8 (DEFAULT_TABLE table=test-table [id=ae45bbb805db4004af6ca6297994075a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:21.198570 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edbdc723a0044e3998928dea71fe03b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:21.198570 10957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edbdc723a0044e3998928dea71fe03b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:21.198611 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.193155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54792 (local address 127.24.255.3:38173)
0926 05:55:21.193516 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:21.193520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:21.193539 (+ 19us) server_negotiation.cc:407] Connection header received
0926 05:55:21.193577 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:21.193579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:21.193593 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:21.193664 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:21.193794 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.195088 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.196389 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.197257 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.198365 (+ 1108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.198478 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:21.198505 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 05:55:21.198547 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":92,"threads_started":1}
I20260926 05:55:21.199138 11088 tablet_service.cc:1467] Processing CreateTablet for tablet edbdc723a0044e3998928dea71fe03b8 (DEFAULT_TABLE table=test-table [id=ae45bbb805db4004af6ca6297994075a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:21.199446 11088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edbdc723a0044e3998928dea71fe03b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:21.202035 11168 tablet_bootstrap.cc:492] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909: Bootstrap starting.
I20260926 05:55:21.202634 11170 tablet_bootstrap.cc:492] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831: Bootstrap starting.
I20260926 05:55:21.202806 11168 tablet_bootstrap.cc:654] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:21.203130 11168 log.cc:824] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:21.203207 11170 tablet_bootstrap.cc:654] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:21.203464 11169 tablet_bootstrap.cc:492] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d: Bootstrap starting.
I20260926 05:55:21.203503 11170 log.cc:824] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:21.203996 11169 tablet_bootstrap.cc:654] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:21.204151 11170 tablet_bootstrap.cc:492] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831: No bootstrap required, opened a new log
I20260926 05:55:21.204205 11170 ts_tablet_manager.cc:1397] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:55:21.204305 11169 log.cc:824] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:21.204838 11168 tablet_bootstrap.cc:492] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909: No bootstrap required, opened a new log
I20260926 05:55:21.205081 11169 tablet_bootstrap.cc:492] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d: No bootstrap required, opened a new log
I20260926 05:55:21.205065 11168 ts_tablet_manager.cc:1397] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 05:55:21.205158 11169 ts_tablet_manager.cc:1397] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:55:21.206627 11170 raft_consensus.cc:348] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4700bea51b59462da74eb364c878b71d" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38315 } } peers { permanent_uuid: "b278530dac2048ca885ecf11f6255909" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37949 } } peers { permanent_uuid: "6ed2de89e81f4149a65f7b341a7e8831" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38173 } }
I20260926 05:55:21.206812 11170 raft_consensus.cc:374] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:21.206846 11170 raft_consensus.cc:729] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed2de89e81f4149a65f7b341a7e8831, State: Initialized, Role: FOLLOWER
I20260926 05:55:21.207006 11170 consensus_queue.cc:260] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4700bea51b59462da74eb364c878b71d" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38315 } } peers { permanent_uuid: "b278530dac2048ca885ecf11f6255909" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37949 } } peers { permanent_uuid: "6ed2de89e81f4149a65f7b341a7e8831" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38173 } }
I20260926 05:55:21.207290 11170 ts_tablet_manager.cc:1428] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:55:21.207513 11154 heartbeater.cc:503] Master 127.24.255.62:38979 was elected leader, sending a full tablet report...
I20260926 05:55:21.207712 11169 raft_consensus.cc:348] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4700bea51b59462da74eb364c878b71d" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38315 } } peers { permanent_uuid: "b278530dac2048ca885ecf11f6255909" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37949 } } peers { permanent_uuid: "6ed2de89e81f4149a65f7b341a7e8831" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38173 } }
I20260926 05:55:21.207897 11169 raft_consensus.cc:374] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:21.207932 11169 raft_consensus.cc:729] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4700bea51b59462da74eb364c878b71d, State: Initialized, Role: FOLLOWER
I20260926 05:55:21.208039 11169 consensus_queue.cc:260] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4700bea51b59462da74eb364c878b71d" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38315 } } peers { permanent_uuid: "b278530dac2048ca885ecf11f6255909" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37949 } } peers { permanent_uuid: "6ed2de89e81f4149a65f7b341a7e8831" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38173 } }
I20260926 05:55:21.207983 11168 raft_consensus.cc:348] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4700bea51b59462da74eb364c878b71d" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38315 } } peers { permanent_uuid: "b278530dac2048ca885ecf11f6255909" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37949 } } peers { permanent_uuid: "6ed2de89e81f4149a65f7b341a7e8831" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38173 } }
I20260926 05:55:21.208168 11168 raft_consensus.cc:374] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:21.208204 11168 raft_consensus.cc:729] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b278530dac2048ca885ecf11f6255909, State: Initialized, Role: FOLLOWER
I20260926 05:55:21.208303 11169 ts_tablet_manager.cc:1428] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:55:21.208433 10892 heartbeater.cc:503] Master 127.24.255.62:38979 was elected leader, sending a full tablet report...
I20260926 05:55:21.208331 11168 consensus_queue.cc:260] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4700bea51b59462da74eb364c878b71d" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38315 } } peers { permanent_uuid: "b278530dac2048ca885ecf11f6255909" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37949 } } peers { permanent_uuid: "6ed2de89e81f4149a65f7b341a7e8831" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38173 } }
I20260926 05:55:21.208891 11168 ts_tablet_manager.cc:1428] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260926 05:55:21.209023 11023 heartbeater.cc:503] Master 127.24.255.62:38979 was elected leader, sending a full tablet report...
W20260926 05:55:21.268118 11024 tablet.cc:2369] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:55:21.304894 11176 raft_consensus.cc:482] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:55:21.305022 11176 raft_consensus.cc:504] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4700bea51b59462da74eb364c878b71d" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38315 } } peers { permanent_uuid: "b278530dac2048ca885ecf11f6255909" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37949 } } peers { permanent_uuid: "6ed2de89e81f4149a65f7b341a7e8831" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38173 } }
I20260926 05:55:21.305418 11176 leader_election.cc:288] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4700bea51b59462da74eb364c878b71d (127.24.255.1:38315), 6ed2de89e81f4149a65f7b341a7e8831 (127.24.255.3:38173)
I20260926 05:55:21.308403 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.305715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:40525 (local address 127.24.255.3:38173)
0926 05:55:21.305758 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:21.305760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:21.305769 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:55:21.305800 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:21.305800 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:21.305809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:21.305855 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:21.306067 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.306647 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.307666 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.308068 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.308144 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.308153 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:21.308293 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 05:55:21.308334 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:55:21.308398 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.305804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:56411 (local address 127.24.255.1:38315)
0926 05:55:21.305860 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:21.305862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:21.305871 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:55:21.305919 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:21.305920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:21.305929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:21.305988 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:21.306197 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.306691 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.307700 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.308068 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.308144 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.308153 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:21.308286 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 05:55:21.308324 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:55:21.308411 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.305618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:38173 (local address 127.24.255.2:40525)
0926 05:55:21.305682 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:21.305694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:21.305743 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:21.305906 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:21.305907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:21.305909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:21.305982 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.305985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.306746 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.306747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.307570 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.307573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.308162 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.308163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.308225 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.308229 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:21.308275 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:55:21.308322 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":19}
I20260926 05:55:21.308437 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.305626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:38315 (local address 127.24.255.2:56411)
0926 05:55:21.305787 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:21.305797 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:21.305851 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:21.305996 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:21.305997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:21.305998 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:21.306067 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.306068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.306786 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.306787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.307616 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:21.307618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.308156 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.308157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:21.308226 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.308229 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:55:21.308276 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 05:55:21.308321 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":10,"thread_start_us":91,"threads_started":1}
I20260926 05:55:21.308836 11108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbdc723a0044e3998928dea71fe03b8" candidate_uuid: "b278530dac2048ca885ecf11f6255909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed2de89e81f4149a65f7b341a7e8831" is_pre_election: true
I20260926 05:55:21.308902 10846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbdc723a0044e3998928dea71fe03b8" candidate_uuid: "b278530dac2048ca885ecf11f6255909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4700bea51b59462da74eb364c878b71d" is_pre_election: true
I20260926 05:55:21.308982 11108 raft_consensus.cc:2454] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b278530dac2048ca885ecf11f6255909 in term 0.
I20260926 05:55:21.309057 10846 raft_consensus.cc:2454] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b278530dac2048ca885ecf11f6255909 in term 0.
I20260926 05:55:21.309232 10913 leader_election.cc:302] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed2de89e81f4149a65f7b341a7e8831, b278530dac2048ca885ecf11f6255909; no voters:
I20260926 05:55:21.309347 11176 raft_consensus.cc:2781] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:55:21.309403 11176 raft_consensus.cc:482] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:55:21.309428 11176 raft_consensus.cc:3037] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:21.310081 11176 raft_consensus.cc:504] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4700bea51b59462da74eb364c878b71d" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38315 } } peers { permanent_uuid: "b278530dac2048ca885ecf11f6255909" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37949 } } peers { permanent_uuid: "6ed2de89e81f4149a65f7b341a7e8831" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38173 } }
I20260926 05:55:21.310225 11176 leader_election.cc:288] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [CANDIDATE]: Term 1 election: Requested vote from peers 4700bea51b59462da74eb364c878b71d (127.24.255.1:38315), 6ed2de89e81f4149a65f7b341a7e8831 (127.24.255.3:38173)
I20260926 05:55:21.310377 10846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbdc723a0044e3998928dea71fe03b8" candidate_uuid: "b278530dac2048ca885ecf11f6255909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4700bea51b59462da74eb364c878b71d"
I20260926 05:55:21.310401 11108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbdc723a0044e3998928dea71fe03b8" candidate_uuid: "b278530dac2048ca885ecf11f6255909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed2de89e81f4149a65f7b341a7e8831"
I20260926 05:55:21.310465 10846 raft_consensus.cc:3037] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:21.310523 11108 raft_consensus.cc:3037] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:21.311358 11108 raft_consensus.cc:2454] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b278530dac2048ca885ecf11f6255909 in term 1.
I20260926 05:55:21.311358 10846 raft_consensus.cc:2454] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b278530dac2048ca885ecf11f6255909 in term 1.
I20260926 05:55:21.311646 10911 leader_election.cc:302] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4700bea51b59462da74eb364c878b71d, b278530dac2048ca885ecf11f6255909; no voters:
I20260926 05:55:21.311746 11176 raft_consensus.cc:2781] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:21.311913 11176 raft_consensus.cc:686] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [term 1 LEADER]: Becoming Leader. State: Replica: b278530dac2048ca885ecf11f6255909, State: Running, Role: LEADER
I20260926 05:55:21.312027 11176 consensus_queue.cc:237] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4700bea51b59462da74eb364c878b71d" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38315 } } peers { permanent_uuid: "b278530dac2048ca885ecf11f6255909" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37949 } } peers { permanent_uuid: "6ed2de89e81f4149a65f7b341a7e8831" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38173 } }
I20260926 05:55:21.312942 10703 catalog_manager.cc:5495] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 reported cstate change: term changed from 0 to 1, leader changed from <none> to b278530dac2048ca885ecf11f6255909 (127.24.255.2). New cstate: current_term: 1 leader_uuid: "b278530dac2048ca885ecf11f6255909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4700bea51b59462da74eb364c878b71d" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b278530dac2048ca885ecf11f6255909" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 37949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ed2de89e81f4149a65f7b341a7e8831" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38173 } health_report { overall_health: UNKNOWN } } }
I20260926 05:55:21.346192 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.343894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.24.255.2:37949)
0926 05:55:21.343950 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:21.343952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:21.343959 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:55:21.343999 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:21.344000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:21.344020 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:21.344104 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:21.344229 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.344703 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.345432 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:21.345620 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:21.345675 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:21.345823 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:21.345927 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:21.346000 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:21.346036 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 05:55:21.346069 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:55:21.348170 10846 raft_consensus.cc:1261] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d [term 1 FOLLOWER]: Refusing update from remote peer b278530dac2048ca885ecf11f6255909: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:21.348170 11108 raft_consensus.cc:1261] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831 [term 1 FOLLOWER]: Refusing update from remote peer b278530dac2048ca885ecf11f6255909: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:21.348628 11176 consensus_queue.cc:1035] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4700bea51b59462da74eb364c878b71d" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 38315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:21.348784 11178 consensus_queue.cc:1035] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed2de89e81f4149a65f7b341a7e8831" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:21.351202 11181 mvcc.cc:205] Tried to move back new op lower bound from 7333487089038925824 to 7333487088894746624. Current Snapshot: MvccSnapshot[applied={T|T < 7333487089038925824}]
I20260926 05:55:21.351400 11182 mvcc.cc:205] Tried to move back new op lower bound from 7333487089038925824 to 7333487088894746624. Current Snapshot: MvccSnapshot[applied={T|T < 7333487089038925824}]
I20260926 05:55:21.351720 11183 mvcc.cc:205] Tried to move back new op lower bound from 7333487089038925824 to 7333487088894746624. Current Snapshot: MvccSnapshot[applied={T|T < 7333487089038925824}]
I20260926 05:55:21.353674 10703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:55:21.353763 10703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:55:21.354568 10703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1f25b9fc171414bbf4a3a48e5d707cd: Sending DeleteTablet for 3 replicas of tablet edbdc723a0044e3998928dea71fe03b8
I20260926 05:55:21.354866 10957 tablet_service.cc:1514] Processing DeleteTablet for tablet edbdc723a0044e3998928dea71fe03b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:21 UTC) from {username='slave'} at 127.0.0.1:41098
I20260926 05:55:21.354945 10826 tablet_service.cc:1514] Processing DeleteTablet for tablet edbdc723a0044e3998928dea71fe03b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:21 UTC) from {username='slave'} at 127.0.0.1:43398
I20260926 05:55:21.355010 11088 tablet_service.cc:1514] Processing DeleteTablet for tablet edbdc723a0044e3998928dea71fe03b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:21 UTC) from {username='slave'} at 127.0.0.1:54792
I20260926 05:55:21.355195 11192 tablet_replica.cc:330] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d: stopping tablet replica
I20260926 05:55:21.355310 11192 raft_consensus.cc:2229] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:55:21.355324 11193 tablet_replica.cc:330] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831: stopping tablet replica
I20260926 05:55:21.355440 11192 raft_consensus.cc:2258] T edbdc723a0044e3998928dea71fe03b8 P 4700bea51b59462da74eb364c878b71d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:21.355473 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 10764
I20260926 05:55:21.355578 11193 raft_consensus.cc:2229] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:55:21.355715 11193 raft_consensus.cc:2258] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:21.356585 11193 ts_tablet_manager.cc:1905] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:55:21.358547 11193 ts_tablet_manager.cc:1918] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:55:21.358631 11193 log.cc:1196] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/edbdc723a0044e3998928dea71fe03b8
I20260926 05:55:21.358860 11193 ts_tablet_manager.cc:1939] T edbdc723a0044e3998928dea71fe03b8 P 6ed2de89e81f4149a65f7b341a7e8831: Deleting consensus metadata
I20260926 05:55:21.359352 10691 catalog_manager.cc:4841] TS 6ed2de89e81f4149a65f7b341a7e8831 (127.24.255.3:38173): tablet edbdc723a0044e3998928dea71fe03b8 (table test-table [id=ae45bbb805db4004af6ca6297994075a]) successfully deleted
W20260926 05:55:21.361822 10689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:38315 (error 108)
I20260926 05:55:21.361948 11191 tablet_replica.cc:330] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909: stopping tablet replica
I20260926 05:55:21.362085 11191 raft_consensus.cc:2229] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:55:21.362246 11191 raft_consensus.cc:2258] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:21.362552 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.362422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:38315 (local address 127.0.0.1:43402)
0926 05:55:21.362470 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:21.362514 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 05:55:21.362552 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 10895
I20260926 05:55:21.362749 11191 ts_tablet_manager.cc:1905] T edbdc723a0044e3998928dea71fe03b8 P b278530dac2048ca885ecf11f6255909: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 05:55:21.363960 10689 catalog_manager.cc:4584] TS 4700bea51b59462da74eb364c878b71d (127.24.255.1:38315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet edbdc723a0044e3998928dea71fe03b8: Network error: Client connection negotiation failed: client connection to 127.24.255.1:38315: connect: Connection refused (error 111)
I20260926 05:55:21.366801 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 11026
I20260926 05:55:21.366873 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:21.366754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:37949 (local address 127.0.0.1:41122)
0926 05:55:21.366807 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:21.366836 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.2:37949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 05:55:21.371300 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 10673
2026-09-26T05:55:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (811 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:21 dist-test-slave-7fr8 krb5kdc[11197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:21 dist-test-slave-7fr8 krb5kdc[11197](info): set up 2 sockets
Sep 26 05:55:21 dist-test-slave-7fr8 krb5kdc[11197](info): commencing operation
krb5kdc: starting...
W20260926 05:55:22.350165 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:22 dist-test-slave-7fr8 krb5kdc[11197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:55:22.473380 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:42919
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:42919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:22.559607 11210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:22.559866 11210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:22.559903 11210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:22.561218 11210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:22.561272 11210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:22.561290 11210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:22.561306 11210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:22.561321 11210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:22.563025 11210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:42919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:42919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:22.563305 11210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:22.563537 11210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:22.569164 11215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:22.569180 11218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:22.569238 11216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:22.569438 11210 server_base.cc:1034] running on GCE node
I20260926 05:55:22.569766 11210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:22.570068 11210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:22.571271 11210 hybrid_clock.cc:648] HybridClock initialized: now 1790402122571250 us; error 32 us; skew 500 ppm
Sep 26 05:55:22 dist-test-slave-7fr8 krb5kdc[11197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402122, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:22.573952 11210 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:55:22.574535 11210 webserver.cc:460] Webserver started at http://127.24.255.62:37013/ using document root <none> and password file <none>
I20260926 05:55:22.574766 11210 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:22.574815 11210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:22.574952 11210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:22.576031 11210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "42d06bcfbebf4954a7d88c0c58edc48a"
format_stamp: "Formatted at 2026-09-26 05:55:22 on dist-test-slave-7fr8"
server_key: "8e668d549b306fae6a3e7b5d90c8e118"
server_key_iv: "d26ef68154ba149578d6139a93963087"
server_key_version: "encryptionkey@0"
I20260926 05:55:22.576372 11210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "42d06bcfbebf4954a7d88c0c58edc48a"
format_stamp: "Formatted at 2026-09-26 05:55:22 on dist-test-slave-7fr8"
server_key: "8e668d549b306fae6a3e7b5d90c8e118"
server_key_iv: "d26ef68154ba149578d6139a93963087"
server_key_version: "encryptionkey@0"
I20260926 05:55:22.577785 11210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:55:22.578680 11225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:22.578843 11210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:55:22.578933 11210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "42d06bcfbebf4954a7d88c0c58edc48a"
format_stamp: "Formatted at 2026-09-26 05:55:22 on dist-test-slave-7fr8"
server_key: "8e668d549b306fae6a3e7b5d90c8e118"
server_key_iv: "d26ef68154ba149578d6139a93963087"
server_key_version: "encryptionkey@0"
I20260926 05:55:22.579010 11210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:22.591979 11210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:22.594987 11210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:22.595185 11210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:22.599197 11210 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:42919
I20260926 05:55:22.599272 11277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:42919 every 8 connection(s)
I20260926 05:55:22.599622 11210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:22.600488 11278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:22.603315 11278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a: Bootstrap starting.
I20260926 05:55:22.603919 11278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:22.604187 11278 log.cc:824] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:22.604950 11278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a: No bootstrap required, opened a new log
I20260926 05:55:22.606703 11278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d06bcfbebf4954a7d88c0c58edc48a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42919 } }
I20260926 05:55:22.606854 11278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:22.606876 11278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d06bcfbebf4954a7d88c0c58edc48a, State: Initialized, Role: FOLLOWER
I20260926 05:55:22.607020 11278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d06bcfbebf4954a7d88c0c58edc48a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42919 } }
I20260926 05:55:22.607095 11278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:22.607235 11278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:22.607290 11278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:22.607920 11278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d06bcfbebf4954a7d88c0c58edc48a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42919 } }
I20260926 05:55:22.608021 11278 leader_election.cc:302] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42d06bcfbebf4954a7d88c0c58edc48a; no voters:
I20260926 05:55:22.608219 11278 leader_election.cc:288] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:22.608271 11281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:22.608423 11281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [term 1 LEADER]: Becoming Leader. State: Replica: 42d06bcfbebf4954a7d88c0c58edc48a, State: Running, Role: LEADER
I20260926 05:55:22.608493 11278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:22.608520 11281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d06bcfbebf4954a7d88c0c58edc48a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42919 } }
I20260926 05:55:22.608861 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 11210
I20260926 05:55:22.608973 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:22.609194 25596 external_mini_cluster.cc:1428] Setting key a44ca77eb11a458440145177bae2cb32
I20260926 05:55:22.609644 11283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42d06bcfbebf4954a7d88c0c58edc48a. Latest consensus state: current_term: 1 leader_uuid: "42d06bcfbebf4954a7d88c0c58edc48a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d06bcfbebf4954a7d88c0c58edc48a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42919 } } }
I20260926 05:55:22.609738 11283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:22.609918 11282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42d06bcfbebf4954a7d88c0c58edc48a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42d06bcfbebf4954a7d88c0c58edc48a" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42919 } } }
I20260926 05:55:22.610168 11282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a [sys.catalog]: This master's current role is: LEADER
W20260926 05:55:22.610388 11299 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:55:22.610435 11299 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:55:22.610488 11298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:22.610651 11298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:22.612361 11298 catalog_manager.cc:1347] Generated new cluster ID: fe67eb054bb4497984d55b893eee8233
I20260926 05:55:22.612419 11298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:55:22 dist-test-slave-7fr8 krb5kdc[11197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:22.619216 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:22.609699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49374 (local address 127.24.255.62:42919)
0926 05:55:22.610123 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:22.610134 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:22.610243 (+ 109us) server_negotiation.cc:407] Connection header received
0926 05:55:22.610382 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:22.610389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:22.610451 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:22.610617 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:22.611369 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:22.612508 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:22.613240 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:22.613679 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:22.613788 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:22.616065 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:22.616071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:22.616071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:22.616084 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:22.617665 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:22.618004 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:22.618007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:22.618007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:22.618144 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:22.618323 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:22.618324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:22.618324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:22.618599 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:22.618745 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:22.618841 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 05:55:22.618988 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":78,"threads_started":1}
I20260926 05:55:22.638923 11298 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:22.639493 11298 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:22.644456 11298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a: Generated new TSK 0
I20260926 05:55:22.644668 11298 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:55:22.678673 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42919
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:22.785735 11304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:22.785966 11304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:22.786000 11304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:22.787279 11304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:22.787443 11304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:22.787547 11304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:55:22.789332 11304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:22.789605 11304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:22.789839 11304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:22.795100 11310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:22.795151 11312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:22.795171 11309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:22.795476 11304 server_base.cc:1034] running on GCE node
I20260926 05:55:22.795781 11304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:22.796043 11304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:22.797225 11304 hybrid_clock.cc:648] HybridClock initialized: now 1790402122797182 us; error 56 us; skew 500 ppm
Sep 26 05:55:22 dist-test-slave-7fr8 krb5kdc[11197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402122, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:22.800210 11304 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:55:22.800747 11304 webserver.cc:460] Webserver started at http://127.24.255.1:36321/ using document root <none> and password file <none>
I20260926 05:55:22.800987 11304 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:22.801040 11304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:22.801172 11304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:22.802306 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "12164d1eb0144200aeef674c504d1bd7"
format_stamp: "Formatted at 2026-09-26 05:55:22 on dist-test-slave-7fr8"
server_key: "c3c67a9ed577fc9da2277f9b956194f0"
server_key_iv: "0cb911e5899827a19179fb7370c97aac"
server_key_version: "encryptionkey@0"
I20260926 05:55:22.802646 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "12164d1eb0144200aeef674c504d1bd7"
format_stamp: "Formatted at 2026-09-26 05:55:22 on dist-test-slave-7fr8"
server_key: "c3c67a9ed577fc9da2277f9b956194f0"
server_key_iv: "0cb911e5899827a19179fb7370c97aac"
server_key_version: "encryptionkey@0"
I20260926 05:55:22.804042 11304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:55:22.804785 11319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:22.804971 11304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:55:22.805058 11304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "12164d1eb0144200aeef674c504d1bd7"
format_stamp: "Formatted at 2026-09-26 05:55:22 on dist-test-slave-7fr8"
server_key: "c3c67a9ed577fc9da2277f9b956194f0"
server_key_iv: "0cb911e5899827a19179fb7370c97aac"
server_key_version: "encryptionkey@0"
I20260926 05:55:22.805122 11304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:22.820348 11304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:22.822606 11304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:22.822749 11304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:22.822969 11304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:22.823287 11304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:22.823320 11304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:22.823350 11304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:22.823371 11304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:22.828826 11304 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:42559
I20260926 05:55:22.828902 11432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:42559 every 8 connection(s)
I20260926 05:55:22.829216 11304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:55:22.834883 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 11304
I20260926 05:55:22.834997 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:55:22.835179 25596 external_mini_cluster.cc:1428] Setting key e9ec50b4ff5dd6b7880d55b1bf4bbeda
Sep 26 05:55:22 dist-test-slave-7fr8 krb5kdc[11197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402122, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:22.838331 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:22.829982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:44383 (local address 127.24.255.62:42919)
0926 05:55:22.830057 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:22.830059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:22.830277 (+ 218us) server_negotiation.cc:407] Connection header received
0926 05:55:22.830795 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:22.830796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:22.830812 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:22.830865 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:22.831817 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:22.832316 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:22.833291 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:22.833498 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:22.833552 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:22.835757 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:22.835758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:22.835759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:22.835770 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:22.837017 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:22.837392 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:22.837393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:22.837393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:22.837442 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:22.837659 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:22.837659 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:22.837660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:22.837773 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:22.837837 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:22.838157 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 05:55:22.838246 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:55:22.838467 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:22.829935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:42919 (local address 127.24.255.1:44383)
0926 05:55:22.830149 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:22.830197 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:22.830690 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:22.830951 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:22.830963 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:22.831523 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:22.831718 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:22.831722 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:22.832398 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:22.832399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:22.833204 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:22.833206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:22.833568 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:22.833569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:22.833698 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:22.834215 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:22.834223 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:22.835668 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:22.837092 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:22.837099 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:22.837101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:22.837328 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:22.837496 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:22.837497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:22.837497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:22.837604 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:22.837846 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:22.837854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:22.837999 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 05:55:22.838123 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 05:55:22.838276 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260926 05:55:22.838934 11433 heartbeater.cc:347] Connected to a master server at 127.24.255.62:42919
I20260926 05:55:22.839025 11433 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:22.839207 11433 heartbeater.cc:511] Master 127.24.255.62:42919 requested a full tablet report, sending...
I20260926 05:55:22.839680 11242 ts_manager.cc:194] Registered new tserver with Master: 12164d1eb0144200aeef674c504d1bd7 (127.24.255.1:42559)
I20260926 05:55:22.840265 11242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:44383
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:55:22.861227 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42919
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:22.943840 11438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:22.944097 11438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:22.944140 11438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:22.945477 11438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:22.945534 11438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:22.945574 11438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:55:22.947396 11438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:22.947788 11438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:22.948067 11438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:22.953218 11443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:22.953230 11444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:22.953431 11446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:22.953603 11438 server_base.cc:1034] running on GCE node
I20260926 05:55:22.953850 11438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:22.954123 11438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:22.955271 11438 hybrid_clock.cc:648] HybridClock initialized: now 1790402122955255 us; error 38 us; skew 500 ppm
Sep 26 05:55:22 dist-test-slave-7fr8 krb5kdc[11197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402122, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:22.958019 11438 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:55:22.958559 11438 webserver.cc:460] Webserver started at http://127.24.255.2:40469/ using document root <none> and password file <none>
I20260926 05:55:22.958829 11438 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:22.958892 11438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:22.959029 11438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:22.960098 11438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45"
format_stamp: "Formatted at 2026-09-26 05:55:22 on dist-test-slave-7fr8"
server_key: "0489437bb69a9d65fa80f96a10e1681d"
server_key_iv: "33bff1676047149fd2da413da8efd9ca"
server_key_version: "encryptionkey@0"
I20260926 05:55:22.960424 11438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45"
format_stamp: "Formatted at 2026-09-26 05:55:22 on dist-test-slave-7fr8"
server_key: "0489437bb69a9d65fa80f96a10e1681d"
server_key_iv: "33bff1676047149fd2da413da8efd9ca"
server_key_version: "encryptionkey@0"
I20260926 05:55:22.962580 11438 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 05:55:22.963824 11453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:22.965068 11438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 05:55:22.965344 11438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45"
format_stamp: "Formatted at 2026-09-26 05:55:22 on dist-test-slave-7fr8"
server_key: "0489437bb69a9d65fa80f96a10e1681d"
server_key_iv: "33bff1676047149fd2da413da8efd9ca"
server_key_version: "encryptionkey@0"
I20260926 05:55:22.965516 11438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:22.980356 11438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:22.982774 11438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:22.982903 11438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:22.983124 11438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:22.983546 11438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:22.983649 11438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:22.983707 11438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:22.983735 11438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:22.991427 11438 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:46087
I20260926 05:55:22.991482 11566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:46087 every 8 connection(s)
I20260926 05:55:22.991806 11438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:55:22.996771 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 11438
I20260926 05:55:22.996896 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:55:22.997088 25596 external_mini_cluster.cc:1428] Setting key 2ea369519cb0b74fd0aad3403acb4237
Sep 26 05:55:22 dist-test-slave-7fr8 krb5kdc[11197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402122, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:23.002138 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:22.992528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:43421 (local address 127.24.255.62:42919)
0926 05:55:22.992595 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:22.992597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:22.992805 (+ 208us) server_negotiation.cc:407] Connection header received
0926 05:55:22.993323 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:22.993325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:22.993337 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:22.993389 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:22.994390 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:22.994946 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:22.995967 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:22.996179 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:22.996237 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:22.998680 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:22.998682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:22.998682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:22.998691 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:23.000381 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:23.000918 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:23.000919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:23.000920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:23.000989 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:23.001273 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:23.001274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:23.001274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:23.001429 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:23.001496 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.001872 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.001986 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:55:23.002270 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:22.992480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:42919 (local address 127.24.255.2:43421)
0926 05:55:22.992687 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:22.992724 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:22.993223 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:22.993458 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:22.993463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:22.994048 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:22.994295 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:22.994300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:22.995042 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:22.995043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:22.995873 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:22.995875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:22.996250 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:22.996251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:22.996381 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:22.996911 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:22.996922 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:22.998577 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:23.000560 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:23.000568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:23.000571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:23.000838 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:23.001091 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:23.001092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:23.001093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:23.001209 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:23.001517 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:23.001523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:23.001696 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 05:55:23.001812 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 05:55:23.001996 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":97,"threads_started":1}
I20260926 05:55:23.002765 11567 heartbeater.cc:347] Connected to a master server at 127.24.255.62:42919
I20260926 05:55:23.002856 11567 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:23.003047 11567 heartbeater.cc:511] Master 127.24.255.62:42919 requested a full tablet report, sending...
I20260926 05:55:23.003396 11242 ts_manager.cc:194] Registered new tserver with Master: 9f9ad00403ad49eb9e6ad2d0eabcdc45 (127.24.255.2:46087)
I20260926 05:55:23.003750 11242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:43421
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:55:23.025504 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42919
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:23.112653 11572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:23.112903 11572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:23.112931 11572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:23.114173 11572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:23.114223 11572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:23.114254 11572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:55:23.115795 11572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:42919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:23.116034 11572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:23.116219 11572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:23.121336 11578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:23.121347 11580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:23.121404 11577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:23.121819 11572 server_base.cc:1034] running on GCE node
I20260926 05:55:23.122012 11572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:23.122289 11572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:23.123450 11572 hybrid_clock.cc:648] HybridClock initialized: now 1790402123123435 us; error 29 us; skew 500 ppm
Sep 26 05:55:23 dist-test-slave-7fr8 krb5kdc[11197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402123, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:23.125995 11572 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:55:23.126505 11572 webserver.cc:460] Webserver started at http://127.24.255.3:42227/ using document root <none> and password file <none>
I20260926 05:55:23.126703 11572 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:23.126744 11572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:23.126838 11572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:23.127847 11572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "f4bd426a5eb945669fb260d100829999"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "7a941b3f2cf17a9a16e40a41be31e816"
server_key_iv: "2a6d8a72ace17824ca5763ae4f102687"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.128172 11572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "f4bd426a5eb945669fb260d100829999"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "7a941b3f2cf17a9a16e40a41be31e816"
server_key_iv: "2a6d8a72ace17824ca5763ae4f102687"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.129431 11572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:55:23.130226 11587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.130378 11572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:23.130460 11572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "f4bd426a5eb945669fb260d100829999"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "7a941b3f2cf17a9a16e40a41be31e816"
server_key_iv: "2a6d8a72ace17824ca5763ae4f102687"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.130522 11572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:23.149866 11572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:23.152004 11572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:23.152122 11572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:23.152312 11572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:23.152623 11572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:23.152657 11572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.152680 11572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:23.152704 11572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.158232 11572 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:39545
I20260926 05:55:23.158300 11700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:39545 every 8 connection(s)
I20260926 05:55:23.158589 11572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:55:23.160807 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 11572
I20260926 05:55:23.160890 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:55:23.161069 25596 external_mini_cluster.cc:1428] Setting key 50be311506db50b03cce206b941bc23c
Sep 26 05:55:23 dist-test-slave-7fr8 krb5kdc[11197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402123, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:23.168025 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.159304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:39363 (local address 127.24.255.62:42919)
0926 05:55:23.159377 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.159379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.159516 (+ 137us) server_negotiation.cc:407] Connection header received
0926 05:55:23.160018 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.160020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.160031 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.160088 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:23.161118 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.161737 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.162821 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.163018 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.163083 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.165317 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:23.165318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:23.165319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:23.165326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:23.166708 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:23.167115 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:23.167116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:23.167116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:23.167169 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:23.167392 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:23.167393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:23.167393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:23.167514 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:23.167570 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.167845 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.167930 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:55:23.168234 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.159266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:42919 (local address 127.24.255.3:39363)
0926 05:55:23.159402 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.159439 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.159915 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.160165 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.160168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.160784 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:23.161026 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.161032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.161812 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.161813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.162740 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.162744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.163086 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.163087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.163222 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.163763 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:23.163771 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:23.165191 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:23.166793 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:23.166801 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:23.166804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:23.167045 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:23.167224 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:23.167224 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:23.167225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:23.167334 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:23.167568 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:23.167575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:23.167719 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 05:55:23.167816 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 05:55:23.167999 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260926 05:55:23.168738 11701 heartbeater.cc:347] Connected to a master server at 127.24.255.62:42919
I20260926 05:55:23.168825 11701 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:23.169020 11701 heartbeater.cc:511] Master 127.24.255.62:42919 requested a full tablet report, sending...
I20260926 05:55:23.169474 11242 ts_manager.cc:194] Registered new tserver with Master: f4bd426a5eb945669fb260d100829999 (127.24.255.3:39545)
I20260926 05:55:23.169864 11242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:39363
I20260926 05:55:23.172984 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:55:23.180979 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.174360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.24.255.62:42919)
0926 05:55:23.174406 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.174408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.174522 (+ 114us) server_negotiation.cc:407] Connection header received
0926 05:55:23.174580 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.174581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.174589 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.174638 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:23.175235 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.175743 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.176501 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.176716 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.176783 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.178172 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:23.178174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:23.178174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:23.178182 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:23.179306 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:23.179937 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:23.179938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:23.179938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:23.180002 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:23.180380 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:23.180381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:23.180381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:23.180536 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:23.180633 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.180807 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.180885 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:55:23.182822 11242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:55:23.184409 11242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:55:23.191233 11715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.186993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.24.255.1:42559)
0926 05:55:23.187141 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.187146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.187445 (+ 299us) server_negotiation.cc:407] Connection header received
0926 05:55:23.187490 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.187492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.187508 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.187573 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:23.187732 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.188835 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.189929 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.190724 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.190773 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.190838 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.191097 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.191144 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260926 05:55:23.191264 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.186804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:46087 (local address 127.0.0.1:50274)
0926 05:55:23.186993 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.187034 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.187100 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.187518 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.187521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.187523 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:23.187625 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.187627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.188742 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.188742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.190026 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.190028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.190891 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.190892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.190985 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.191001 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 05:55:23.191089 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 05:55:23.191137 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260926 05:55:23.191515 11717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.187178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.24.255.2:46087)
0926 05:55:23.187368 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.187373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.187384 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:55:23.187423 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.187426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.187440 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.187504 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:23.187698 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.188641 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.190135 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.190809 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.190876 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.190928 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.191389 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.191441 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260926 05:55:23.191267 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.186803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:42559 (local address 127.0.0.1:50730)
0926 05:55:23.186993 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.187382 (+ 389us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.187424 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.187645 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.187646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.187647 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:23.187680 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.187681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.188977 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.188978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.189819 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.189821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.190910 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.190911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.191000 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.191004 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:23.191065 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:55:23.191129 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":2,"spinlock_wait_cycles":896,"thread_start_us":98,"threads_started":1}
I20260926 05:55:23.192073 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.186806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:39545 (local address 127.0.0.1:41204)
0926 05:55:23.187259 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.187268 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.187323 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.187574 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.187575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.187577 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:23.187625 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.187627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.189955 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.189956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.190715 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.190718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.191738 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.191739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.191798 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.191803 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:55:23.191964 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 05:55:23.192008 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":2,"thread_start_us":118,"threads_started":1}
I20260926 05:55:23.192282 11501 tablet_service.cc:1467] Processing CreateTablet for tablet f796af217c924bffbb8f3d2cba3ef4a0 (DEFAULT_TABLE table=test-table [id=a2e3c1bddbe24865afed0006fbb635f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:23.192443 11716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.187095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.24.255.3:39545)
0926 05:55:23.187252 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.187256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.187313 (+ 57us) server_negotiation.cc:407] Connection header received
0926 05:55:23.187386 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.187388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.187403 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.187484 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:23.188760 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.189835 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.190798 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.191661 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.191721 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.191776 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.192326 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.192375 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260926 05:55:23.192672 11501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f796af217c924bffbb8f3d2cba3ef4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:23.193056 11635 tablet_service.cc:1467] Processing CreateTablet for tablet f796af217c924bffbb8f3d2cba3ef4a0 (DEFAULT_TABLE table=test-table [id=a2e3c1bddbe24865afed0006fbb635f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:23.193347 11635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f796af217c924bffbb8f3d2cba3ef4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:23.194600 11367 tablet_service.cc:1467] Processing CreateTablet for tablet f796af217c924bffbb8f3d2cba3ef4a0 (DEFAULT_TABLE table=test-table [id=a2e3c1bddbe24865afed0006fbb635f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:23.194873 11367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f796af217c924bffbb8f3d2cba3ef4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:23.196099 11718 tablet_bootstrap.cc:492] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45: Bootstrap starting.
I20260926 05:55:23.196444 11719 tablet_bootstrap.cc:492] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999: Bootstrap starting.
I20260926 05:55:23.196935 11718 tablet_bootstrap.cc:654] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:23.196960 11719 tablet_bootstrap.cc:654] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:23.197382 11719 log.cc:824] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:23.197495 11718 log.cc:824] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:23.197955 11722 tablet_bootstrap.cc:492] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7: Bootstrap starting.
I20260926 05:55:23.198153 11719 tablet_bootstrap.cc:492] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999: No bootstrap required, opened a new log
I20260926 05:55:23.198225 11719 ts_tablet_manager.cc:1397] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:55:23.198249 11718 tablet_bootstrap.cc:492] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45: No bootstrap required, opened a new log
I20260926 05:55:23.198310 11718 ts_tablet_manager.cc:1397] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:55:23.198719 11722 tablet_bootstrap.cc:654] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:23.199280 11722 log.cc:824] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:23.200065 11722 tablet_bootstrap.cc:492] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7: No bootstrap required, opened a new log
I20260926 05:55:23.199970 11719 raft_consensus.cc:348] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46087 } } peers { permanent_uuid: "12164d1eb0144200aeef674c504d1bd7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 42559 } } peers { permanent_uuid: "f4bd426a5eb945669fb260d100829999" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39545 } }
I20260926 05:55:23.200111 11719 raft_consensus.cc:374] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:23.200137 11722 ts_tablet_manager.cc:1397] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:55:23.200140 11719 raft_consensus.cc:729] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4bd426a5eb945669fb260d100829999, State: Initialized, Role: FOLLOWER
I20260926 05:55:23.200286 11719 consensus_queue.cc:260] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46087 } } peers { permanent_uuid: "12164d1eb0144200aeef674c504d1bd7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 42559 } } peers { permanent_uuid: "f4bd426a5eb945669fb260d100829999" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39545 } }
I20260926 05:55:23.200227 11718 raft_consensus.cc:348] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46087 } } peers { permanent_uuid: "12164d1eb0144200aeef674c504d1bd7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 42559 } } peers { permanent_uuid: "f4bd426a5eb945669fb260d100829999" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39545 } }
I20260926 05:55:23.200376 11718 raft_consensus.cc:374] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:23.200433 11718 raft_consensus.cc:729] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9ad00403ad49eb9e6ad2d0eabcdc45, State: Initialized, Role: FOLLOWER
I20260926 05:55:23.200565 11719 ts_tablet_manager.cc:1428] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:55:23.200554 11718 consensus_queue.cc:260] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46087 } } peers { permanent_uuid: "12164d1eb0144200aeef674c504d1bd7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 42559 } } peers { permanent_uuid: "f4bd426a5eb945669fb260d100829999" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39545 } }
I20260926 05:55:23.200675 11701 heartbeater.cc:503] Master 127.24.255.62:42919 was elected leader, sending a full tablet report...
I20260926 05:55:23.200831 11567 heartbeater.cc:503] Master 127.24.255.62:42919 was elected leader, sending a full tablet report...
I20260926 05:55:23.200796 11718 ts_tablet_manager.cc:1428] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:55:23.201973 11722 raft_consensus.cc:348] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46087 } } peers { permanent_uuid: "12164d1eb0144200aeef674c504d1bd7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 42559 } } peers { permanent_uuid: "f4bd426a5eb945669fb260d100829999" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39545 } }
I20260926 05:55:23.202193 11722 raft_consensus.cc:374] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:23.202242 11722 raft_consensus.cc:729] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12164d1eb0144200aeef674c504d1bd7, State: Initialized, Role: FOLLOWER
I20260926 05:55:23.202346 11722 consensus_queue.cc:260] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46087 } } peers { permanent_uuid: "12164d1eb0144200aeef674c504d1bd7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 42559 } } peers { permanent_uuid: "f4bd426a5eb945669fb260d100829999" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39545 } }
I20260926 05:55:23.202571 11722 ts_tablet_manager.cc:1428] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:55:23.202611 11433 heartbeater.cc:503] Master 127.24.255.62:42919 was elected leader, sending a full tablet report...
I20260926 05:55:23.221895 11725 raft_consensus.cc:482] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:55:23.222033 11725 raft_consensus.cc:504] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46087 } } peers { permanent_uuid: "12164d1eb0144200aeef674c504d1bd7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 42559 } } peers { permanent_uuid: "f4bd426a5eb945669fb260d100829999" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39545 } }
I20260926 05:55:23.222442 11725 leader_election.cc:288] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12164d1eb0144200aeef674c504d1bd7 (127.24.255.1:42559), f4bd426a5eb945669fb260d100829999 (127.24.255.3:39545)
I20260926 05:55:23.226766 11716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.222735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:50101 (local address 127.24.255.3:39545)
0926 05:55:23.222826 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.222828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.222836 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:55:23.222865 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.222866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.222875 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.222921 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:23.223270 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.224201 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.225727 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.226356 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.226445 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.226454 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.226641 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.226686 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 05:55:23.226761 11715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.222902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:47117 (local address 127.24.255.1:42559)
0926 05:55:23.222953 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.222955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.222965 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:55:23.223027 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.223028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.223038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.223090 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:23.223274 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.224203 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.225727 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.226356 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.226439 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.226450 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.226640 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.226686 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 05:55:23.226775 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.222647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:39545 (local address 127.24.255.2:50101)
0926 05:55:23.222701 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.222712 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.222753 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.223069 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.223070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.223072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:23.223162 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.223164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.224311 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.224312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.225611 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.225614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.226453 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.226454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.226550 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.226556 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:55:23.226621 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 05:55:23.226671 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":38}
I20260926 05:55:23.226780 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.222667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:42559 (local address 127.24.255.2:47117)
0926 05:55:23.222898 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.222909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.222968 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.223143 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.223143 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.223145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:23.223197 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.223198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.224311 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.224312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.225614 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.225616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.226453 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.226454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.226572 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.226575 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:55:23.226625 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 05:55:23.226670 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":86,"thread_start_us":120,"threads_started":1}
I20260926 05:55:23.227161 11655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796af217c924bffbb8f3d2cba3ef4a0" candidate_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4bd426a5eb945669fb260d100829999" is_pre_election: true
I20260926 05:55:23.227311 11655 raft_consensus.cc:2454] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9ad00403ad49eb9e6ad2d0eabcdc45 in term 0.
I20260926 05:55:23.227406 11387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796af217c924bffbb8f3d2cba3ef4a0" candidate_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12164d1eb0144200aeef674c504d1bd7" is_pre_election: true
I20260926 05:55:23.227555 11454 leader_election.cc:302] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f9ad00403ad49eb9e6ad2d0eabcdc45, f4bd426a5eb945669fb260d100829999; no voters:
I20260926 05:55:23.227587 11387 raft_consensus.cc:2454] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9ad00403ad49eb9e6ad2d0eabcdc45 in term 0.
I20260926 05:55:23.227694 11725 raft_consensus.cc:2781] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:55:23.227767 11725 raft_consensus.cc:482] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:55:23.227799 11725 raft_consensus.cc:3037] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:23.228540 11725 raft_consensus.cc:504] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46087 } } peers { permanent_uuid: "12164d1eb0144200aeef674c504d1bd7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 42559 } } peers { permanent_uuid: "f4bd426a5eb945669fb260d100829999" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39545 } }
I20260926 05:55:23.228679 11725 leader_election.cc:288] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [CANDIDATE]: Term 1 election: Requested vote from peers 12164d1eb0144200aeef674c504d1bd7 (127.24.255.1:42559), f4bd426a5eb945669fb260d100829999 (127.24.255.3:39545)
I20260926 05:55:23.228847 11387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796af217c924bffbb8f3d2cba3ef4a0" candidate_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12164d1eb0144200aeef674c504d1bd7"
I20260926 05:55:23.228868 11655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796af217c924bffbb8f3d2cba3ef4a0" candidate_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4bd426a5eb945669fb260d100829999"
I20260926 05:55:23.228924 11387 raft_consensus.cc:3037] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:23.228938 11655 raft_consensus.cc:3037] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:23.229621 11387 raft_consensus.cc:2454] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9ad00403ad49eb9e6ad2d0eabcdc45 in term 1.
I20260926 05:55:23.229664 11655 raft_consensus.cc:2454] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9ad00403ad49eb9e6ad2d0eabcdc45 in term 1.
I20260926 05:55:23.229779 11456 leader_election.cc:302] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12164d1eb0144200aeef674c504d1bd7, 9f9ad00403ad49eb9e6ad2d0eabcdc45; no voters:
I20260926 05:55:23.229861 11725 raft_consensus.cc:2781] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:23.230029 11725 raft_consensus.cc:686] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [term 1 LEADER]: Becoming Leader. State: Replica: 9f9ad00403ad49eb9e6ad2d0eabcdc45, State: Running, Role: LEADER
I20260926 05:55:23.230154 11725 consensus_queue.cc:237] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46087 } } peers { permanent_uuid: "12164d1eb0144200aeef674c504d1bd7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 42559 } } peers { permanent_uuid: "f4bd426a5eb945669fb260d100829999" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39545 } }
I20260926 05:55:23.231011 11241 catalog_manager.cc:5495] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f9ad00403ad49eb9e6ad2d0eabcdc45 (127.24.255.2). New cstate: current_term: 1 leader_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ad00403ad49eb9e6ad2d0eabcdc45" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 46087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12164d1eb0144200aeef674c504d1bd7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 42559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4bd426a5eb945669fb260d100829999" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39545 } health_report { overall_health: UNKNOWN } } }
I20260926 05:55:23.237030 11717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.234814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.24.255.2:46087)
0926 05:55:23.234857 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.234859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.234868 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:55:23.234940 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.234941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.234959 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.235012 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:23.235148 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.235624 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.236326 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.236513 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.236567 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.236713 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:23.236804 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:23.236888 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.236937 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.236947 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:55:23.238938 11387 raft_consensus.cc:1261] T f796af217c924bffbb8f3d2cba3ef4a0 P 12164d1eb0144200aeef674c504d1bd7 [term 1 FOLLOWER]: Refusing update from remote peer 9f9ad00403ad49eb9e6ad2d0eabcdc45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:23.238938 11655 raft_consensus.cc:1261] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999 [term 1 FOLLOWER]: Refusing update from remote peer 9f9ad00403ad49eb9e6ad2d0eabcdc45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:23.239266 11725 consensus_queue.cc:1035] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4bd426a5eb945669fb260d100829999" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:23.239490 11728 consensus_queue.cc:1035] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12164d1eb0144200aeef674c504d1bd7" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 42559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:23.242337 11733 mvcc.cc:205] Tried to move back new op lower bound from 7333487096783626240 to 7333487096751357952. Current Snapshot: MvccSnapshot[applied={T|T < 7333487096783626240}]
I20260926 05:55:23.242767 11732 mvcc.cc:205] Tried to move back new op lower bound from 7333487096783626240 to 7333487096751357952. Current Snapshot: MvccSnapshot[applied={T|T < 7333487096783626240}]
I20260926 05:55:23.244951 11241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:55:23.245049 11241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:55:23.245833 11241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42d06bcfbebf4954a7d88c0c58edc48a: Sending DeleteTablet for 3 replicas of tablet f796af217c924bffbb8f3d2cba3ef4a0
I20260926 05:55:23.246248 11501 tablet_service.cc:1514] Processing DeleteTablet for tablet f796af217c924bffbb8f3d2cba3ef4a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:23 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:50274
I20260926 05:55:23.246275 11635 tablet_service.cc:1514] Processing DeleteTablet for tablet f796af217c924bffbb8f3d2cba3ef4a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:23 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:41204
I20260926 05:55:23.246529 11741 tablet_replica.cc:330] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999: stopping tablet replica
I20260926 05:55:23.246526 11367 tablet_service.cc:1514] Processing DeleteTablet for tablet f796af217c924bffbb8f3d2cba3ef4a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:23 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:50730
I20260926 05:55:23.246660 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 11304
I20260926 05:55:23.246671 11742 tablet_replica.cc:330] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45: stopping tablet replica
I20260926 05:55:23.246800 11742 raft_consensus.cc:2229] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:55:23.246896 11741 raft_consensus.cc:2229] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:55:23.247036 11742 raft_consensus.cc:2258] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:23.247156 11741 raft_consensus.cc:2258] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:23.247982 11742 ts_tablet_manager.cc:1905] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:55:23.249547 11741 ts_tablet_manager.cc:1905] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:55:23.249627 11742 ts_tablet_manager.cc:1918] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:55:23.249691 11742 log.cc:1196] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/f796af217c924bffbb8f3d2cba3ef4a0
I20260926 05:55:23.249949 11742 ts_tablet_manager.cc:1939] T f796af217c924bffbb8f3d2cba3ef4a0 P 9f9ad00403ad49eb9e6ad2d0eabcdc45: Deleting consensus metadata
I20260926 05:55:23.250411 11227 catalog_manager.cc:4841] TS 9f9ad00403ad49eb9e6ad2d0eabcdc45 (127.24.255.2:46087): tablet f796af217c924bffbb8f3d2cba3ef4a0 (table test-table [id=a2e3c1bddbe24865afed0006fbb635f7]) successfully deleted
I20260926 05:55:23.251224 11741 ts_tablet_manager.cc:1918] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:55:23.251283 11741 log.cc:1196] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/f796af217c924bffbb8f3d2cba3ef4a0
I20260926 05:55:23.251493 11741 ts_tablet_manager.cc:1939] T f796af217c924bffbb8f3d2cba3ef4a0 P f4bd426a5eb945669fb260d100829999: Deleting consensus metadata
I20260926 05:55:23.251987 11226 catalog_manager.cc:4841] TS f4bd426a5eb945669fb260d100829999 (127.24.255.3:39545): tablet f796af217c924bffbb8f3d2cba3ef4a0 (table test-table [id=a2e3c1bddbe24865afed0006fbb635f7]) successfully deleted
W20260926 05:55:23.257136 11228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:42559 (error 108)
I20260926 05:55:23.257218 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 11438
I20260926 05:55:23.258250 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.258064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:42559 (local address 127.0.0.1:50734)
0926 05:55:23.258152 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.258206 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:42559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 05:55:23.258399 11228 catalog_manager.cc:4584] TS 12164d1eb0144200aeef674c504d1bd7 (127.24.255.1:42559): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f796af217c924bffbb8f3d2cba3ef4a0: Network error: Client connection negotiation failed: client connection to 127.24.255.1:42559: connect: Connection refused (error 111)
I20260926 05:55:23.262288 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 11572
I20260926 05:55:23.267066 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 11210
2026-09-26T05:55:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1896 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T05:55:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:23Z Disabled control of system clock
I20260926 05:55:23.294800 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:43177
--webserver_interface=127.24.255.62
--webserver_port=0
--builtin_ntp_servers=127.24.255.20:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:43177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:55:23.377404 11748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:23.377655 11748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:23.377687 11748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:23.378960 11748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:23.379011 11748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:23.379027 11748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:23.379043 11748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:23.379060 11748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:23.380540 11748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:43177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:43177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:23.380802 11748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:23.381093 11748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:23.385722 11754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:23.385733 11756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:23.385733 11753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:23.386030 11748 server_base.cc:1034] running on GCE node
I20260926 05:55:23.386276 11748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:23.386598 11748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:23.387763 11748 hybrid_clock.cc:648] HybridClock initialized: now 1790402123387743 us; error 39 us; skew 500 ppm
I20260926 05:55:23.389076 11748 webserver.cc:460] Webserver started at http://127.24.255.62:38023/ using document root <none> and password file <none>
I20260926 05:55:23.389312 11748 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:23.389361 11748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:23.389489 11748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:23.390662 11748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "ec3a5c494f6d4a2989c198a4dac3fc9c"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "665bb51e63efeb9b235de8c1eb5d2e9b"
server_key_iv: "55bbd6c17ad3dd1f337c0f06a1faa322"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.390987 11748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "ec3a5c494f6d4a2989c198a4dac3fc9c"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "665bb51e63efeb9b235de8c1eb5d2e9b"
server_key_iv: "55bbd6c17ad3dd1f337c0f06a1faa322"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.392251 11748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:55:23.392949 11762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.393106 11748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:55:23.393188 11748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "ec3a5c494f6d4a2989c198a4dac3fc9c"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "665bb51e63efeb9b235de8c1eb5d2e9b"
server_key_iv: "55bbd6c17ad3dd1f337c0f06a1faa322"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.393252 11748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:23.417306 11748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:23.417641 11748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:23.417771 11748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:23.421718 11748 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:43177
I20260926 05:55:23.421753 11814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:43177 every 8 connection(s)
I20260926 05:55:23.422114 11748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:23.423056 11815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:23.425650 11815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c: Bootstrap starting.
I20260926 05:55:23.426311 11815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:23.426604 11815 log.cc:824] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:23.427750 11815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c: No bootstrap required, opened a new log
I20260926 05:55:23.430536 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 11748
I20260926 05:55:23.430718 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:23.430977 25596 external_mini_cluster.cc:1428] Setting key 4c719f3449c5c1b10977c2ebc17704b1
I20260926 05:55:23.431423 11815 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3a5c494f6d4a2989c198a4dac3fc9c" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 43177 } }
I20260926 05:55:23.431713 11815 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:23.431766 11815 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3a5c494f6d4a2989c198a4dac3fc9c, State: Initialized, Role: FOLLOWER
I20260926 05:55:23.431931 11815 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3a5c494f6d4a2989c198a4dac3fc9c" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 43177 } }
I20260926 05:55:23.432026 11815 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:23.432062 11815 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:23.432117 11815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:23.433050 11815 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3a5c494f6d4a2989c198a4dac3fc9c" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 43177 } }
I20260926 05:55:23.433260 11815 leader_election.cc:302] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec3a5c494f6d4a2989c198a4dac3fc9c; no voters:
I20260926 05:55:23.433553 11815 leader_election.cc:288] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:23.433596 11820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:23.433707 11820 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [term 1 LEADER]: Becoming Leader. State: Replica: ec3a5c494f6d4a2989c198a4dac3fc9c, State: Running, Role: LEADER
I20260926 05:55:23.433866 11815 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:23.433835 11820 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3a5c494f6d4a2989c198a4dac3fc9c" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 43177 } }
I20260926 05:55:23.434252 11820 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec3a5c494f6d4a2989c198a4dac3fc9c. Latest consensus state: current_term: 1 leader_uuid: "ec3a5c494f6d4a2989c198a4dac3fc9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3a5c494f6d4a2989c198a4dac3fc9c" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 43177 } } }
I20260926 05:55:23.434334 11820 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:23.434484 11821 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec3a5c494f6d4a2989c198a4dac3fc9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3a5c494f6d4a2989c198a4dac3fc9c" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 43177 } } }
I20260926 05:55:23.434558 11821 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:23.435344 11826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:23.435861 11826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:23.438125 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.431746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.24.255.62:43177)
0926 05:55:23.432028 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.432037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.432074 (+ 37us) server_negotiation.cc:407] Connection header received
0926 05:55:23.432231 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.432237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.432300 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.432436 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:23.432728 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.434255 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.435430 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.435992 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.436124 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.437233 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:23.437237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:23.437238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:23.437445 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:23.437494 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:23.437513 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:23.437793 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.437821 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.437905 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":133,"threads_started":1}
I20260926 05:55:23.438423 11826 catalog_manager.cc:1347] Generated new cluster ID: 06444e608da842c0a0e454408acc0df1
I20260926 05:55:23.438483 11826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:23.450333 11826 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:23.451076 11826 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:23.461009 11826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c: Generated new TSK 0
I20260926 05:55:23.461248 11826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:23.467484 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:43177
--builtin_ntp_servers=127.24.255.20:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:55:23.555245 11839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:23.555469 11839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:23.555500 11839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:23.556919 11839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:23.557037 11839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:23.557080 11839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:55:23.559903 11839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:43177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:23.560225 11839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:23.560567 11839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:23.565914 11847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:23.565964 11845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:23.565915 11844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:23.566308 11839 server_base.cc:1034] running on GCE node
I20260926 05:55:23.566517 11839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:23.566782 11839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:23.567981 11839 hybrid_clock.cc:648] HybridClock initialized: now 1790402123567961 us; error 32 us; skew 500 ppm
I20260926 05:55:23.569892 11839 webserver.cc:460] Webserver started at http://127.24.255.1:36987/ using document root <none> and password file <none>
I20260926 05:55:23.570402 11839 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:23.570502 11839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:23.570638 11839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:23.571772 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "ba334dec0d58406c812c723d8afc90fa"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "31c1fe0a278e590131b141a626741e06"
server_key_iv: "ffa1b6d75fef75bc833de037e77269e3"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.572139 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "ba334dec0d58406c812c723d8afc90fa"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "31c1fe0a278e590131b141a626741e06"
server_key_iv: "ffa1b6d75fef75bc833de037e77269e3"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.573498 11839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:55:23.574273 11853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.574537 11839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:23.574657 11839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "ba334dec0d58406c812c723d8afc90fa"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "31c1fe0a278e590131b141a626741e06"
server_key_iv: "ffa1b6d75fef75bc833de037e77269e3"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.574739 11839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:23.588155 11839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:23.588452 11839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:23.588557 11839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:23.588768 11839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:23.589133 11839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:23.589167 11839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.589196 11839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:23.589213 11839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.594769 11839 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:36195
I20260926 05:55:23.594877 11966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:36195 every 8 connection(s)
I20260926 05:55:23.595175 11839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:55:23.603739 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 11839
I20260926 05:55:23.603837 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:55:23.604066 25596 external_mini_cluster.cc:1428] Setting key 1bebd4200da4732b1b9b6b8c0c5e342c
I20260926 05:55:23.605423 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:43177
--builtin_ntp_servers=127.24.255.20:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 05:55:23.607930 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.596010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:57017 (local address 127.24.255.62:43177)
0926 05:55:23.596098 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.596100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.596420 (+ 320us) server_negotiation.cc:407] Connection header received
0926 05:55:23.597337 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.597445 (+ 108us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.597478 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.597587 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:23.598563 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.599165 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.600469 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.600703 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.600772 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.607620 (+ 6848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:23.607622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:23.607623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:23.607633 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:23.607653 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:23.607657 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:23.607700 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.607787 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.607838 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 05:55:23.608196 11969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.595941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:43177 (local address 127.24.255.1:57017)
0926 05:55:23.596208 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.596277 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.597142 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.597762 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.597767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.597779 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:23.598404 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.598411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.599292 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.599293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.600317 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.600322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.600819 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.600820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.600953 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.607371 (+ 6418us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:55:23.607372 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:23.607420 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:55:23.607458 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:23.607712 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:23.607719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:23.607723 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:23.607787 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:55:23.607928 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260926 05:55:23.608691 11967 heartbeater.cc:347] Connected to a master server at 127.24.255.62:43177
I20260926 05:55:23.608791 11967 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:23.608999 11967 heartbeater.cc:511] Master 127.24.255.62:43177 requested a full tablet report, sending...
I20260926 05:55:23.609840 11779 ts_manager.cc:194] Registered new tserver with Master: ba334dec0d58406c812c723d8afc90fa (127.24.255.1:36195)
I20260926 05:55:23.610692 11779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.255.1:57017
W20260926 05:55:23.690025 11970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:23.690289 11970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:23.690322 11970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:23.691831 11970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:23.691900 11970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:23.691946 11970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:55:23.693456 11970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:43177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:23.693719 11970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:23.694017 11970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:23.699021 11976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:23.699010 11975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:23.699239 11970 server_base.cc:1034] running on GCE node
W20260926 05:55:23.699213 11978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:23.699570 11970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:23.699821 11970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:23.701006 11970 hybrid_clock.cc:648] HybridClock initialized: now 1790402123700989 us; error 32 us; skew 500 ppm
I20260926 05:55:23.702486 11970 webserver.cc:460] Webserver started at http://127.24.255.2:41031/ using document root <none> and password file <none>
I20260926 05:55:23.702715 11970 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:23.702771 11970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:23.702896 11970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:23.704064 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "0f86104b2ea8480084be00ef0166105d"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "4190edd1701be12fb42c804f48f5a793"
server_key_iv: "3ef7463edd7f2deaedfe9d015c57b6f6"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.704418 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "0f86104b2ea8480084be00ef0166105d"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "4190edd1701be12fb42c804f48f5a793"
server_key_iv: "3ef7463edd7f2deaedfe9d015c57b6f6"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.705747 11970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:55:23.706552 11984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.706709 11970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.706791 11970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "0f86104b2ea8480084be00ef0166105d"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "4190edd1701be12fb42c804f48f5a793"
server_key_iv: "3ef7463edd7f2deaedfe9d015c57b6f6"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.706856 11970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:23.717110 11970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:23.717422 11970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:23.717543 11970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:23.717751 11970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:23.718142 11970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:23.718189 11970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.718219 11970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:23.718243 11970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.724121 11970 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:35307
I20260926 05:55:23.724192 12097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:35307 every 8 connection(s)
I20260926 05:55:23.724502 11970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:55:23.729770 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.725282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:54253 (local address 127.24.255.62:43177)
0926 05:55:23.725359 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.725362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.725552 (+ 190us) server_negotiation.cc:407] Connection header received
0926 05:55:23.726149 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.726150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.726161 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.726212 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:23.726864 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.727460 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.728431 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.728622 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.728676 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.729495 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:23.729497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:23.729497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:23.729503 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:23.729519 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:23.729523 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:23.729570 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.729651 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.729683 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:55:23.729985 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.725220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:43177 (local address 127.24.255.2:54253)
0926 05:55:23.725438 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.725473 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.726013 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.726280 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.726284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.726294 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:23.726780 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.726785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.727547 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.727549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.728350 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.728353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.728690 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.728691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.728819 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.729343 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:55:23.729343 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:23.729373 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:55:23.729398 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:23.729580 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:23.729586 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:23.729589 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:55:23.729650 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:55:23.729762 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260926 05:55:23.730669 12098 heartbeater.cc:347] Connected to a master server at 127.24.255.62:43177
I20260926 05:55:23.730762 12098 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:23.730917 12098 heartbeater.cc:511] Master 127.24.255.62:43177 requested a full tablet report, sending...
I20260926 05:55:23.731268 11779 ts_manager.cc:194] Registered new tserver with Master: 0f86104b2ea8480084be00ef0166105d (127.24.255.2:35307)
I20260926 05:55:23.731640 11779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.255.2:54253
I20260926 05:55:23.732438 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 11970
I20260926 05:55:23.732558 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:55:23.732717 25596 external_mini_cluster.cc:1428] Setting key 6bbac7fb5a31cb059e06aa6562df8db9
I20260926 05:55:23.733613 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:43177
--builtin_ntp_servers=127.24.255.20:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:55:23.817215 12101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:23.817430 12101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:23.817451 12101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:23.818670 12101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:23.818711 12101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:23.818737 12101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:55:23.820065 12101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:43177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:23.820272 12101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:23.820542 12101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:23.825659 12109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:23.825673 12106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:23.825838 12101 server_base.cc:1034] running on GCE node
W20260926 05:55:23.825707 12107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:23.826418 12101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:23.826745 12101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:23.827916 12101 hybrid_clock.cc:648] HybridClock initialized: now 1790402123827900 us; error 32 us; skew 500 ppm
I20260926 05:55:23.829344 12101 webserver.cc:460] Webserver started at http://127.24.255.3:36037/ using document root <none> and password file <none>
I20260926 05:55:23.829581 12101 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:23.829639 12101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:23.829770 12101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:23.830897 12101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "48786f23607d48f28a776ac72009072d"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "9cfa271695790c93001cc71c216aff84"
server_key_iv: "0b643415c20cd9c2657e6abe20c8bd1e"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.831230 12101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "48786f23607d48f28a776ac72009072d"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "9cfa271695790c93001cc71c216aff84"
server_key_iv: "0b643415c20cd9c2657e6abe20c8bd1e"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.832495 12101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:55:23.833307 12115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.833500 12101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:23.833583 12101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "48786f23607d48f28a776ac72009072d"
format_stamp: "Formatted at 2026-09-26 05:55:23 on dist-test-slave-7fr8"
server_key: "9cfa271695790c93001cc71c216aff84"
server_key_iv: "0b643415c20cd9c2657e6abe20c8bd1e"
server_key_version: "encryptionkey@0"
I20260926 05:55:23.833679 12101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:23.850509 12101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:23.850828 12101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:23.850956 12101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:23.851181 12101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:23.851509 12101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:23.851543 12101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.851574 12101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:23.851591 12101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:23.858590 12101 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:37953
I20260926 05:55:23.858656 12228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:37953 every 8 connection(s)
I20260926 05:55:23.858990 12101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:55:23.864332 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.859714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:36757 (local address 127.24.255.62:43177)
0926 05:55:23.859777 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.859779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.860085 (+ 306us) server_negotiation.cc:407] Connection header received
0926 05:55:23.860578 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.860579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.860592 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.860653 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:23.861346 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.861860 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.862899 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.863123 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.863179 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.864040 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:23.864042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:23.864042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:23.864048 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:23.864072 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:23.864076 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:23.864120 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.864207 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.864244 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:55:23.864534 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.859758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:43177 (local address 127.24.255.3:36757)
0926 05:55:23.859941 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.859981 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.860478 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.860726 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.860729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.860738 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:23.861251 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.861255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.861953 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.861954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.862806 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.862809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.863200 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.863201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.863332 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.863889 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:55:23.863889 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:23.863919 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:55:23.863947 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:23.864130 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:23.864136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:23.864139 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:55:23.864203 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:55:23.864319 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260926 05:55:23.865145 12229 heartbeater.cc:347] Connected to a master server at 127.24.255.62:43177
I20260926 05:55:23.865259 12229 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:23.865420 12229 heartbeater.cc:511] Master 127.24.255.62:43177 requested a full tablet report, sending...
I20260926 05:55:23.865792 11779 ts_manager.cc:194] Registered new tserver with Master: 48786f23607d48f28a776ac72009072d (127.24.255.3:37953)
I20260926 05:55:23.866190 11779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.255.3:36757
I20260926 05:55:23.868865 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 12101
I20260926 05:55:23.868957 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:55:23.869114 25596 external_mini_cluster.cc:1428] Setting key b6d00d3cbf5326b92a36ed360b40d5ae
I20260926 05:55:23.870373 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:55:23.874815 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.871867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58052 (local address 127.24.255.62:43177)
0926 05:55:23.871933 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.871935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.871977 (+ 42us) server_negotiation.cc:407] Connection header received
0926 05:55:23.872058 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.872059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.872071 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.872133 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:23.872298 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.872800 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.873514 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.873717 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.873771 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.874518 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:23.874520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:23.874520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:23.874525 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:23.874542 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:23.874546 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:23.874627 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.874711 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.874723 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:55:23.876830 11779 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:55:23.878590 11779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:55:23.885406 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.880986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:35307 (local address 127.0.0.1:50134)
0926 05:55:23.881247 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.881255 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.881294 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.881487 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.881490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.881492 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:23.881569 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.881571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.882796 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.882797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.883959 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.883962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.885072 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.885079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.885170 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.885185 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 05:55:23.885261 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 05:55:23.885321 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":35,"thread_start_us":97,"threads_started":1}
I20260926 05:55:23.885421 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.881082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.24.255.2:35307)
0926 05:55:23.881287 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.881292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.881305 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:55:23.881398 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.881400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.881412 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.881472 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:23.881647 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.882715 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.884070 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.884921 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.885017 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.885185 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.885280 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.885321 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260926 05:55:23.885936 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.880984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:36195 (local address 127.0.0.1:40012)
0926 05:55:23.881179 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.881219 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.881290 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.881620 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.881621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.881623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:23.881654 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.881656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.882859 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.882860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.884221 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.884224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.885396 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.885397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.885454 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.885458 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:23.885486 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 05:55:23.885850 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260926 05:55:23.886430 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.881087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.24.255.1:36195)
0926 05:55:23.881375 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.881378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.881453 (+ 75us) server_negotiation.cc:407] Connection header received
0926 05:55:23.881506 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.881508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.881524 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.881618 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:23.881709 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.882742 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.884318 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.885290 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.886148 (+ 858us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.886251 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.886279 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.886322 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":76,"threads_started":1}
I20260926 05:55:23.886407 12032 tablet_service.cc:1467] Processing CreateTablet for tablet 9a55a1bec7c440d3852ad9902172cdba (DEFAULT_TABLE table=test-table [id=070bc55b2e6f423e9d3c156470ba6637]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:23.886641 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.881523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.24.255.3:37953)
0926 05:55:23.882292 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.882296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.882306 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:55:23.882342 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.882343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.882358 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.882430 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:23.882568 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.883693 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.885451 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.886229 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.886296 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.886367 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:23.886457 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 05:55:23.886498 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":91,"threads_started":1}
I20260926 05:55:23.886801 12032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a55a1bec7c440d3852ad9902172cdba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:23.886917 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.881203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:37953 (local address 127.0.0.1:59358)
0926 05:55:23.881709 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.881720 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.881765 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.882454 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.882455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.882457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:23.882501 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.882503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.883810 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.883811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.885345 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.885348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.886300 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.886301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:23.886368 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:23.886372 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:23.886440 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 05:55:23.886657 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":72,"threads_started":1}
I20260926 05:55:23.887526 12163 tablet_service.cc:1467] Processing CreateTablet for tablet 9a55a1bec7c440d3852ad9902172cdba (DEFAULT_TABLE table=test-table [id=070bc55b2e6f423e9d3c156470ba6637]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:23.887841 12163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a55a1bec7c440d3852ad9902172cdba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:23.889528 12243 tablet_bootstrap.cc:492] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d: Bootstrap starting.
I20260926 05:55:23.890292 12243 tablet_bootstrap.cc:654] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:23.890445 12244 tablet_bootstrap.cc:492] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d: Bootstrap starting.
I20260926 05:55:23.890239 11901 tablet_service.cc:1467] Processing CreateTablet for tablet 9a55a1bec7c440d3852ad9902172cdba (DEFAULT_TABLE table=test-table [id=070bc55b2e6f423e9d3c156470ba6637]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:23.890712 12243 log.cc:824] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:23.890759 11901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a55a1bec7c440d3852ad9902172cdba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:23.891269 12244 tablet_bootstrap.cc:654] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:23.891716 12244 log.cc:824] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:23.891760 12243 tablet_bootstrap.cc:492] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d: No bootstrap required, opened a new log
I20260926 05:55:23.891860 12243 ts_tablet_manager.cc:1397] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:55:23.894438 12247 tablet_bootstrap.cc:492] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa: Bootstrap starting.
I20260926 05:55:23.894526 12243 raft_consensus.cc:348] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba334dec0d58406c812c723d8afc90fa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 36195 } } peers { permanent_uuid: "48786f23607d48f28a776ac72009072d" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 37953 } } peers { permanent_uuid: "0f86104b2ea8480084be00ef0166105d" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35307 } }
I20260926 05:55:23.894717 12243 raft_consensus.cc:374] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:23.894761 12243 raft_consensus.cc:729] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f86104b2ea8480084be00ef0166105d, State: Initialized, Role: FOLLOWER
I20260926 05:55:23.894922 12243 consensus_queue.cc:260] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba334dec0d58406c812c723d8afc90fa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 36195 } } peers { permanent_uuid: "48786f23607d48f28a776ac72009072d" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 37953 } } peers { permanent_uuid: "0f86104b2ea8480084be00ef0166105d" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35307 } }
I20260926 05:55:23.895015 12244 tablet_bootstrap.cc:492] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d: No bootstrap required, opened a new log
I20260926 05:55:23.895089 12244 ts_tablet_manager.cc:1397] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 05:55:23.895159 12247 tablet_bootstrap.cc:654] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:23.895195 12243 ts_tablet_manager.cc:1428] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260926 05:55:23.895346 12098 heartbeater.cc:503] Master 127.24.255.62:43177 was elected leader, sending a full tablet report...
I20260926 05:55:23.895463 12247 log.cc:824] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:23.896281 12247 tablet_bootstrap.cc:492] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa: No bootstrap required, opened a new log
I20260926 05:55:23.896348 12247 ts_tablet_manager.cc:1397] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:55:23.897224 12244 raft_consensus.cc:348] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba334dec0d58406c812c723d8afc90fa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 36195 } } peers { permanent_uuid: "48786f23607d48f28a776ac72009072d" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 37953 } } peers { permanent_uuid: "0f86104b2ea8480084be00ef0166105d" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35307 } }
I20260926 05:55:23.897476 12244 raft_consensus.cc:374] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:23.897526 12244 raft_consensus.cc:729] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48786f23607d48f28a776ac72009072d, State: Initialized, Role: FOLLOWER
I20260926 05:55:23.897650 12244 consensus_queue.cc:260] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba334dec0d58406c812c723d8afc90fa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 36195 } } peers { permanent_uuid: "48786f23607d48f28a776ac72009072d" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 37953 } } peers { permanent_uuid: "0f86104b2ea8480084be00ef0166105d" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35307 } }
I20260926 05:55:23.897927 12244 ts_tablet_manager.cc:1428] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:55:23.897969 12229 heartbeater.cc:503] Master 127.24.255.62:43177 was elected leader, sending a full tablet report...
I20260926 05:55:23.898576 12247 raft_consensus.cc:348] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba334dec0d58406c812c723d8afc90fa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 36195 } } peers { permanent_uuid: "48786f23607d48f28a776ac72009072d" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 37953 } } peers { permanent_uuid: "0f86104b2ea8480084be00ef0166105d" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35307 } }
I20260926 05:55:23.898743 12247 raft_consensus.cc:374] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:23.898784 12247 raft_consensus.cc:729] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba334dec0d58406c812c723d8afc90fa, State: Initialized, Role: FOLLOWER
I20260926 05:55:23.898898 12247 consensus_queue.cc:260] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba334dec0d58406c812c723d8afc90fa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 36195 } } peers { permanent_uuid: "48786f23607d48f28a776ac72009072d" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 37953 } } peers { permanent_uuid: "0f86104b2ea8480084be00ef0166105d" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35307 } }
I20260926 05:55:23.899151 12247 ts_tablet_manager.cc:1428] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:55:23.899277 11967 heartbeater.cc:503] Master 127.24.255.62:43177 was elected leader, sending a full tablet report...
W20260926 05:55:23.975158 12099 tablet.cc:2369] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:55:23.997601 12248 raft_consensus.cc:482] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:55:23.997748 12248 raft_consensus.cc:504] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba334dec0d58406c812c723d8afc90fa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 36195 } } peers { permanent_uuid: "48786f23607d48f28a776ac72009072d" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 37953 } } peers { permanent_uuid: "0f86104b2ea8480084be00ef0166105d" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35307 } }
I20260926 05:55:23.998189 12248 leader_election.cc:288] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba334dec0d58406c812c723d8afc90fa (127.24.255.1:36195), 48786f23607d48f28a776ac72009072d (127.24.255.3:37953)
I20260926 05:55:24.001087 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.998403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:36195 (local address 127.24.255.2:54243)
0926 05:55:23.998459 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.998473 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.998539 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.998728 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.998729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.998731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:23.998797 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.998800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.999539 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.999540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:24.000306 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:24.000308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:24.000842 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:24.000843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:24.000908 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:24.000911 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:55:24.000957 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:55:24.001005 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":12}
I20260926 05:55:24.001085 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.998542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:54243 (local address 127.24.255.1:36195)
0926 05:55:23.998585 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.998587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.998594 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:55:23.998621 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.998622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.998633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.998679 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:23.998967 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.999434 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:24.000397 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:24.000765 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:24.000827 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:24.000837 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:24.000976 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 05:55:24.001010 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:55:24.001479 11921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a55a1bec7c440d3852ad9902172cdba" candidate_uuid: "0f86104b2ea8480084be00ef0166105d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba334dec0d58406c812c723d8afc90fa" is_pre_election: true
I20260926 05:55:24.001575 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.998556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:37953 (local address 127.24.255.2:60687)
0926 05:55:23.998731 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:23.998741 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:23.998792 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:23.998914 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:23.998915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:23.998917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:23.998983 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:23.998985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.999559 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:23.999560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:24.000733 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:24.000736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:24.001332 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:24.001334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:24.001410 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:24.001414 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:24.001460 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:55:24.001499 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":106,"threads_started":1}
I20260926 05:55:24.001591 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:23.998677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:60687 (local address 127.24.255.3:37953)
0926 05:55:23.998716 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:23.998718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:23.998788 (+ 70us) server_negotiation.cc:407] Connection header received
0926 05:55:23.998849 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:23.998850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:23.998858 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:23.998902 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:23.999046 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:23.999489 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:24.000849 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:24.001245 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:24.001313 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:24.001320 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:24.001481 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 05:55:24.001534 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:55:24.001637 11921 raft_consensus.cc:2454] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f86104b2ea8480084be00ef0166105d in term 0.
I20260926 05:55:24.001875 11987 leader_election.cc:302] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f86104b2ea8480084be00ef0166105d, ba334dec0d58406c812c723d8afc90fa; no voters:
I20260926 05:55:24.001983 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a55a1bec7c440d3852ad9902172cdba" candidate_uuid: "0f86104b2ea8480084be00ef0166105d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48786f23607d48f28a776ac72009072d" is_pre_election: true
I20260926 05:55:24.002024 12248 raft_consensus.cc:2781] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:55:24.002069 12248 raft_consensus.cc:482] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:55:24.002090 12248 raft_consensus.cc:3037] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:24.002144 12183 raft_consensus.cc:2454] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f86104b2ea8480084be00ef0166105d in term 0.
I20260926 05:55:24.002889 12248 raft_consensus.cc:504] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba334dec0d58406c812c723d8afc90fa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 36195 } } peers { permanent_uuid: "48786f23607d48f28a776ac72009072d" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 37953 } } peers { permanent_uuid: "0f86104b2ea8480084be00ef0166105d" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35307 } }
I20260926 05:55:24.003012 12248 leader_election.cc:288] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [CANDIDATE]: Term 1 election: Requested vote from peers ba334dec0d58406c812c723d8afc90fa (127.24.255.1:36195), 48786f23607d48f28a776ac72009072d (127.24.255.3:37953)
I20260926 05:55:24.003196 11921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a55a1bec7c440d3852ad9902172cdba" candidate_uuid: "0f86104b2ea8480084be00ef0166105d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba334dec0d58406c812c723d8afc90fa"
I20260926 05:55:24.003273 11921 raft_consensus.cc:3037] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:24.003427 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a55a1bec7c440d3852ad9902172cdba" candidate_uuid: "0f86104b2ea8480084be00ef0166105d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48786f23607d48f28a776ac72009072d"
I20260926 05:55:24.003521 12183 raft_consensus.cc:3037] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:24.004062 11921 raft_consensus.cc:2454] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f86104b2ea8480084be00ef0166105d in term 1.
I20260926 05:55:24.004253 11987 leader_election.cc:302] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f86104b2ea8480084be00ef0166105d, ba334dec0d58406c812c723d8afc90fa; no voters:
I20260926 05:55:24.004251 12183 raft_consensus.cc:2454] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f86104b2ea8480084be00ef0166105d in term 1.
I20260926 05:55:24.004361 12248 raft_consensus.cc:2781] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:24.004554 12248 raft_consensus.cc:686] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [term 1 LEADER]: Becoming Leader. State: Replica: 0f86104b2ea8480084be00ef0166105d, State: Running, Role: LEADER
I20260926 05:55:24.004665 12248 consensus_queue.cc:237] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba334dec0d58406c812c723d8afc90fa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 36195 } } peers { permanent_uuid: "48786f23607d48f28a776ac72009072d" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 37953 } } peers { permanent_uuid: "0f86104b2ea8480084be00ef0166105d" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35307 } }
I20260926 05:55:24.005613 11779 catalog_manager.cc:5495] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f86104b2ea8480084be00ef0166105d (127.24.255.2). New cstate: current_term: 1 leader_uuid: "0f86104b2ea8480084be00ef0166105d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba334dec0d58406c812c723d8afc90fa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 36195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48786f23607d48f28a776ac72009072d" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 37953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f86104b2ea8480084be00ef0166105d" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 35307 } health_report { overall_health: HEALTHY } } }
I20260926 05:55:24.032776 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:24.030022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50144 (local address 127.24.255.2:35307)
0926 05:55:24.030078 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:24.030080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:24.030088 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:55:24.030157 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:24.030157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:24.030178 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:24.030239 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:24.030582 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:24.031061 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:24.031878 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:24.032087 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:24.032158 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:24.032447 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:24.032553 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:24.032636 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:24.032684 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 05:55:24.032694 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:55:24.034708 12183 raft_consensus.cc:1261] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d [term 1 FOLLOWER]: Refusing update from remote peer 0f86104b2ea8480084be00ef0166105d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:24.034749 11921 raft_consensus.cc:1261] T 9a55a1bec7c440d3852ad9902172cdba P ba334dec0d58406c812c723d8afc90fa [term 1 FOLLOWER]: Refusing update from remote peer 0f86104b2ea8480084be00ef0166105d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:24.035081 12248 consensus_queue.cc:1035] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba334dec0d58406c812c723d8afc90fa" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 36195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:24.035224 12253 consensus_queue.cc:1035] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [LEADER]: Connected to new peer: Peer: permanent_uuid: "48786f23607d48f28a776ac72009072d" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 37953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:24.037878 12256 mvcc.cc:205] Tried to move back new op lower bound from 7333487100043132928 to 7333487099923771392. Current Snapshot: MvccSnapshot[applied={T|T < 7333487100043132928}]
I20260926 05:55:24.038606 12258 mvcc.cc:205] Tried to move back new op lower bound from 7333487100043132928 to 7333487099923771392. Current Snapshot: MvccSnapshot[applied={T|T < 7333487100043132928}]
I20260926 05:55:24.040580 11779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:55:24.040670 11779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:55:24.041406 11779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec3a5c494f6d4a2989c198a4dac3fc9c: Sending DeleteTablet for 3 replicas of tablet 9a55a1bec7c440d3852ad9902172cdba
I20260926 05:55:24.041828 11901 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a55a1bec7c440d3852ad9902172cdba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:24 UTC) from {username='slave'} at 127.0.0.1:40012
I20260926 05:55:24.041850 12163 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a55a1bec7c440d3852ad9902172cdba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:24 UTC) from {username='slave'} at 127.0.0.1:59358
I20260926 05:55:24.042153 12268 tablet_replica.cc:330] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d: stopping tablet replica
I20260926 05:55:24.042128 12032 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a55a1bec7c440d3852ad9902172cdba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:24 UTC) from {username='slave'} at 127.0.0.1:50134
I20260926 05:55:24.042191 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 11839
I20260926 05:55:24.042263 12268 raft_consensus.cc:2229] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:55:24.042325 12269 tablet_replica.cc:330] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d: stopping tablet replica
I20260926 05:55:24.042362 12268 raft_consensus.cc:2258] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:24.042399 12269 raft_consensus.cc:2229] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:55:24.042610 12269 raft_consensus.cc:2258] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:24.043085 12268 ts_tablet_manager.cc:1905] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:55:24.043205 12269 ts_tablet_manager.cc:1905] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:55:24.045086 12268 ts_tablet_manager.cc:1918] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:55:24.045101 12269 ts_tablet_manager.cc:1918] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:55:24.045161 12268 log.cc:1196] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/9a55a1bec7c440d3852ad9902172cdba
I20260926 05:55:24.045171 12269 log.cc:1196] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/9a55a1bec7c440d3852ad9902172cdba
I20260926 05:55:24.045413 12268 ts_tablet_manager.cc:1939] T 9a55a1bec7c440d3852ad9902172cdba P 48786f23607d48f28a776ac72009072d: Deleting consensus metadata
I20260926 05:55:24.045413 12269 ts_tablet_manager.cc:1939] T 9a55a1bec7c440d3852ad9902172cdba P 0f86104b2ea8480084be00ef0166105d: Deleting consensus metadata
I20260926 05:55:24.045879 11763 catalog_manager.cc:4841] TS 0f86104b2ea8480084be00ef0166105d (127.24.255.2:35307): tablet 9a55a1bec7c440d3852ad9902172cdba (table test-table [id=070bc55b2e6f423e9d3c156470ba6637]) successfully deleted
I20260926 05:55:24.046443 11765 catalog_manager.cc:4841] TS 48786f23607d48f28a776ac72009072d (127.24.255.3:37953): tablet 9a55a1bec7c440d3852ad9902172cdba (table test-table [id=070bc55b2e6f423e9d3c156470ba6637]) successfully deleted
W20260926 05:55:24.049757 11765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:36195 (error 108)
I20260926 05:55:24.050401 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 11970
I20260926 05:55:24.052349 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:24.052185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:36195 (local address 127.0.0.1:40016)
0926 05:55:24.052254 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:24.052308 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:36195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 05:55:24.052529 11765 catalog_manager.cc:4584] TS ba334dec0d58406c812c723d8afc90fa (127.24.255.1:36195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a55a1bec7c440d3852ad9902172cdba: Network error: Client connection negotiation failed: client connection to 127.24.255.1:36195: connect: Connection refused (error 111)
I20260926 05:55:24.055397 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 12101
I20260926 05:55:24.060047 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 11748
2026-09-26T05:55:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (793 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:24 dist-test-slave-7fr8 krb5kdc[12273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:24 dist-test-slave-7fr8 krb5kdc[12273](info): set up 2 sockets
Sep 26 05:55:24 dist-test-slave-7fr8 krb5kdc[12273](info): commencing operation
krb5kdc: starting...
W20260926 05:55:25.026258 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:25 dist-test-slave-7fr8 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:55:25.148940 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:33115
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:33115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:25.231473 12286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:25.231714 12286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:25.231750 12286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:25.232951 12286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:25.233008 12286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:25.233029 12286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:25.233049 12286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:25.233068 12286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:25.234666 12286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:33115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:33115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:25.234942 12286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:25.235172 12286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:25.240193 12292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:25.240268 12291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:25.240274 12286 server_base.cc:1034] running on GCE node
W20260926 05:55:25.240193 12294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:25.240653 12286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:25.240938 12286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:25.242148 12286 hybrid_clock.cc:648] HybridClock initialized: now 1790402125242126 us; error 34 us; skew 500 ppm
Sep 26 05:55:25 dist-test-slave-7fr8 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402125, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:25.244678 12286 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:55:25.245157 12286 webserver.cc:460] Webserver started at http://127.24.255.62:46739/ using document root <none> and password file <none>
I20260926 05:55:25.245349 12286 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:25.245389 12286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:25.245476 12286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:25.246699 12286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "ce2ebeccd6cb42d788eff4525f53fe50"
format_stamp: "Formatted at 2026-09-26 05:55:25 on dist-test-slave-7fr8"
server_key: "79fa22aba0a13ecffc1d9620276295e2"
server_key_iv: "37cb2175ee83991c5592a17bb967a688"
server_key_version: "encryptionkey@0"
I20260926 05:55:25.247051 12286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "ce2ebeccd6cb42d788eff4525f53fe50"
format_stamp: "Formatted at 2026-09-26 05:55:25 on dist-test-slave-7fr8"
server_key: "79fa22aba0a13ecffc1d9620276295e2"
server_key_iv: "37cb2175ee83991c5592a17bb967a688"
server_key_version: "encryptionkey@0"
I20260926 05:55:25.248413 12286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:55:25.249207 12301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:25.249408 12286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:25.249500 12286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "ce2ebeccd6cb42d788eff4525f53fe50"
format_stamp: "Formatted at 2026-09-26 05:55:25 on dist-test-slave-7fr8"
server_key: "79fa22aba0a13ecffc1d9620276295e2"
server_key_iv: "37cb2175ee83991c5592a17bb967a688"
server_key_version: "encryptionkey@0"
I20260926 05:55:25.249570 12286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:25.274981 12286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:25.277333 12286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:25.277454 12286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:25.281371 12286 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:33115
I20260926 05:55:25.281463 12353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:33115 every 8 connection(s)
I20260926 05:55:25.281774 12286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:25.283185 12354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:25.284420 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 12286
I20260926 05:55:25.284552 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:25.284756 25596 external_mini_cluster.cc:1428] Setting key 53d008818a8b14e5d637bc0a0d48bfc8
I20260926 05:55:25.285895 12354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50: Bootstrap starting.
I20260926 05:55:25.286610 12354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:25.286932 12354 log.cc:824] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:25.287629 12354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50: No bootstrap required, opened a new log
I20260926 05:55:25.289875 12354 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2ebeccd6cb42d788eff4525f53fe50" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33115 } }
I20260926 05:55:25.290051 12354 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:25.290078 12354 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2ebeccd6cb42d788eff4525f53fe50, State: Initialized, Role: FOLLOWER
I20260926 05:55:25.290235 12354 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2ebeccd6cb42d788eff4525f53fe50" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33115 } }
I20260926 05:55:25.290319 12354 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:25.290354 12354 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:25.290400 12354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:25.290975 12354 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2ebeccd6cb42d788eff4525f53fe50" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33115 } }
I20260926 05:55:25.291079 12354 leader_election.cc:302] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce2ebeccd6cb42d788eff4525f53fe50; no voters:
I20260926 05:55:25.291266 12354 leader_election.cc:288] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:25.291302 12359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:25.291467 12359 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [term 1 LEADER]: Becoming Leader. State: Replica: ce2ebeccd6cb42d788eff4525f53fe50, State: Running, Role: LEADER
I20260926 05:55:25.291553 12354 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:25.291576 12359 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2ebeccd6cb42d788eff4525f53fe50" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33115 } }
Sep 26 05:55:25 dist-test-slave-7fr8 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:25.291929 12359 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce2ebeccd6cb42d788eff4525f53fe50. Latest consensus state: current_term: 1 leader_uuid: "ce2ebeccd6cb42d788eff4525f53fe50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2ebeccd6cb42d788eff4525f53fe50" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33115 } } }
I20260926 05:55:25.292012 12359 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:25.291939 12361 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce2ebeccd6cb42d788eff4525f53fe50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2ebeccd6cb42d788eff4525f53fe50" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33115 } } }
I20260926 05:55:25.292157 12361 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:25.292284 12365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:25.293594 12365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:25.295414 12365 catalog_manager.cc:1347] Generated new cluster ID: bca6baab4efe4fa58516b01c639eaf2c
I20260926 05:55:25.295468 12365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:25.296633 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.285427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.24.255.62:33115)
0926 05:55:25.285593 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:25.285601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:25.285628 (+ 27us) server_negotiation.cc:407] Connection header received
0926 05:55:25.286074 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:25.286081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:25.286169 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:25.286286 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:25.287112 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.288098 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.289185 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.289613 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.289697 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.292435 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:25.292441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:25.292441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:25.292453 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:25.294554 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:25.295053 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:25.295057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:25.295058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:25.295167 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:25.295354 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:25.295355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:25.295355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:25.295665 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:25.295801 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:25.296283 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 05:55:25.296420 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260926 05:55:25.307478 12365 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:25.308496 12365 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:25.313743 12365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50: Generated new TSK 0
I20260926 05:55:25.313973 12365 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.255.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
Entry for principal kudu/127.24.255.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab.
I20260926 05:55:25.342213 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:33115
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:25.425606 12380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:25.425843 12380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:25.425869 12380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:25.427299 12380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:25.427356 12380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:25.427387 12380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:55:25.428987 12380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.1.keytab
--principal=kudu/127.24.255.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:33115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:25.429349 12380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:25.429633 12380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:25.434811 12388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:25.434810 12385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:25.435196 12386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:25.435261 12380 server_base.cc:1034] running on GCE node
I20260926 05:55:25.435634 12380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:25.435918 12380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:25.437103 12380 hybrid_clock.cc:648] HybridClock initialized: now 1790402125437085 us; error 31 us; skew 500 ppm
Sep 26 05:55:25 dist-test-slave-7fr8 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402125, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:25.440079 12380 init.cc:377] Logged in from keytab as kudu/127.24.255.1@KRBTEST.COM (short username kudu)
I20260926 05:55:25.440553 12380 webserver.cc:460] Webserver started at http://127.24.255.1:43743/ using document root <none> and password file <none>
I20260926 05:55:25.440757 12380 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:25.440806 12380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:25.440934 12380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:25.442202 12380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6"
format_stamp: "Formatted at 2026-09-26 05:55:25 on dist-test-slave-7fr8"
server_key: "0ef2a46b6702ab1cfb52ee9413c1b2b6"
server_key_iv: "4ae4354c4afb6a4690eda85e77a49870"
server_key_version: "encryptionkey@0"
I20260926 05:55:25.442572 12380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6"
format_stamp: "Formatted at 2026-09-26 05:55:25 on dist-test-slave-7fr8"
server_key: "0ef2a46b6702ab1cfb52ee9413c1b2b6"
server_key_iv: "4ae4354c4afb6a4690eda85e77a49870"
server_key_version: "encryptionkey@0"
I20260926 05:55:25.444015 12380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:55:25.444790 12395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:25.445070 12380 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:55:25.445165 12380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6"
format_stamp: "Formatted at 2026-09-26 05:55:25 on dist-test-slave-7fr8"
server_key: "0ef2a46b6702ab1cfb52ee9413c1b2b6"
server_key_iv: "4ae4354c4afb6a4690eda85e77a49870"
server_key_version: "encryptionkey@0"
I20260926 05:55:25.445228 12380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:25.473747 12380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:25.476051 12380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:25.476199 12380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:25.476450 12380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:25.476755 12380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:25.476788 12380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:25.476818 12380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:25.476835 12380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:25.482445 12380 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:41707
I20260926 05:55:25.482504 12508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:41707 every 8 connection(s)
I20260926 05:55:25.482805 12380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:55:25.487607 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 12380
I20260926 05:55:25.487720 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:55:25.487936 25596 external_mini_cluster.cc:1428] Setting key 24d88e414d288136d178c4be39eb989c
Sep 26 05:55:25 dist-test-slave-7fr8 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402125, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.1@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:25.492846 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.483470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:33115 (local address 127.24.255.1:47133)
0926 05:55:25.483671 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:25.483712 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:25.484163 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:25.484400 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:25.484403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:25.484986 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:25.485187 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.485191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.485855 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.485855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.486656 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.486658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.487045 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.487046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.487161 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.487687 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:25.487698 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:25.489310 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:25.491056 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:25.491065 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:25.491068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:25.491369 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:25.491572 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:25.491573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:25.491573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:25.491712 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:25.492223 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:25.492230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:25.492365 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 05:55:25.492496 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 05:55:25.492675 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":70,"threads_started":1}
I20260926 05:55:25.493269 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.483623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:47133 (local address 127.24.255.62:33115)
0926 05:55:25.483681 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:25.483683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:25.483774 (+ 91us) server_negotiation.cc:407] Connection header received
0926 05:55:25.484266 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:25.484267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:25.484278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:25.484339 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:25.485273 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.485786 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.486732 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.486949 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.487011 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.489401 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:25.489402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:25.489402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:25.489409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:25.490982 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:25.491442 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:25.491443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:25.491444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:25.491503 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:25.491795 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:25.491796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:25.491796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:25.491924 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:25.492184 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:25.493104 (+ 920us) server_negotiation.cc:299] Negotiation successful
0926 05:55:25.493182 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:55:25.493641 12509 heartbeater.cc:347] Connected to a master server at 127.24.255.62:33115
I20260926 05:55:25.493736 12509 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:25.493889 12509 heartbeater.cc:511] Master 127.24.255.62:33115 requested a full tablet report, sending...
I20260926 05:55:25.494436 12318 ts_manager.cc:194] Registered new tserver with Master: d0c8c3e221b94a5d9f448d5c5a6ae6c6 (127.24.255.1:41707)
I20260926 05:55:25.495049 12318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.1@KRBTEST.COM'} at 127.24.255.1:47133
WARNING: no policy specified for kudu/127.24.255.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
Entry for principal kudu/127.24.255.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab.
I20260926 05:55:25.514290 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:33115
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:25.596916 12514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:25.597141 12514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:25.597164 12514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:25.598393 12514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:25.598439 12514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:25.598467 12514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:55:25.599964 12514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.2.keytab
--principal=kudu/127.24.255.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:33115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:25.600222 12514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:25.600446 12514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:25.605386 12522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:25.605382 12519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:25.605422 12514 server_base.cc:1034] running on GCE node
W20260926 05:55:25.605386 12520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:25.605755 12514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:25.605994 12514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:25.607157 12514 hybrid_clock.cc:648] HybridClock initialized: now 1790402125607101 us; error 73 us; skew 500 ppm
Sep 26 05:55:25 dist-test-slave-7fr8 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402125, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:25.609750 12514 init.cc:377] Logged in from keytab as kudu/127.24.255.2@KRBTEST.COM (short username kudu)
I20260926 05:55:25.610404 12514 webserver.cc:460] Webserver started at http://127.24.255.2:37115/ using document root <none> and password file <none>
I20260926 05:55:25.610623 12514 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:25.610672 12514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:25.610798 12514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:25.611882 12514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "d3b36afb8bcb4c278f915aff2b493779"
format_stamp: "Formatted at 2026-09-26 05:55:25 on dist-test-slave-7fr8"
server_key: "59f2054b47b0133ddda4982dc9a81bcd"
server_key_iv: "ca6077356adcde4973b877b4e31f5a6d"
server_key_version: "encryptionkey@0"
I20260926 05:55:25.612223 12514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "d3b36afb8bcb4c278f915aff2b493779"
format_stamp: "Formatted at 2026-09-26 05:55:25 on dist-test-slave-7fr8"
server_key: "59f2054b47b0133ddda4982dc9a81bcd"
server_key_iv: "ca6077356adcde4973b877b4e31f5a6d"
server_key_version: "encryptionkey@0"
I20260926 05:55:25.613533 12514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:55:25.614341 12529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:25.614478 12514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:55:25.614569 12514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "d3b36afb8bcb4c278f915aff2b493779"
format_stamp: "Formatted at 2026-09-26 05:55:25 on dist-test-slave-7fr8"
server_key: "59f2054b47b0133ddda4982dc9a81bcd"
server_key_iv: "ca6077356adcde4973b877b4e31f5a6d"
server_key_version: "encryptionkey@0"
I20260926 05:55:25.614629 12514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:25.633519 12514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:25.635941 12514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:25.636104 12514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:25.636358 12514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:25.636691 12514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:25.636729 12514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:25.636765 12514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:25.636792 12514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:25.643074 12514 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:43505
I20260926 05:55:25.643136 12642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:43505 every 8 connection(s)
I20260926 05:55:25.643486 12514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:55:25 dist-test-slave-7fr8 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402125, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.2@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:25.649699 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 12514
I20260926 05:55:25.649802 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:55:25.650004 25596 external_mini_cluster.cc:1428] Setting key 73d82f616d9a3917f78eb207e38231e7
I20260926 05:55:25.653026 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.644283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:36571 (local address 127.24.255.62:33115)
0926 05:55:25.644341 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:25.644343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:25.644544 (+ 201us) server_negotiation.cc:407] Connection header received
0926 05:55:25.645050 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:25.645052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:25.645064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:25.645107 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:25.646150 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.646647 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.647634 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.647850 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.647912 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.650341 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:25.650342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:25.650343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:25.650352 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:25.651680 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:25.652074 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:25.652075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:25.652075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:25.652129 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:25.652352 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:25.652352 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:25.652353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:25.652499 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:25.652553 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:25.652844 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 05:55:25.652931 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:55:25.653231 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.644252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:33115 (local address 127.24.255.2:36571)
0926 05:55:25.644426 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:25.644462 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:25.644954 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:25.645185 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:25.645189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:25.645805 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:25.646012 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.646017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.646732 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.646733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.647550 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.647553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.647960 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.647961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.648080 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.648633 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:25.648642 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:25.650238 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:25.651748 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:25.651757 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:25.651760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:25.652005 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:25.652186 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:25.652187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:25.652187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:25.652298 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:25.652558 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:25.652565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:25.652727 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 05:55:25.652817 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 05:55:25.653002 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260926 05:55:25.653713 12643 heartbeater.cc:347] Connected to a master server at 127.24.255.62:33115
I20260926 05:55:25.653815 12643 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:25.654009 12643 heartbeater.cc:511] Master 127.24.255.62:33115 requested a full tablet report, sending...
I20260926 05:55:25.654662 12318 ts_manager.cc:194] Registered new tserver with Master: d3b36afb8bcb4c278f915aff2b493779 (127.24.255.2:43505)
I20260926 05:55:25.655196 12318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.2@KRBTEST.COM'} at 127.24.255.2:36571
WARNING: no policy specified for kudu/127.24.255.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
Entry for principal kudu/127.24.255.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab.
I20260926 05:55:25.678139 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:33115
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:25.760841 12648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:25.761068 12648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:25.761093 12648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:25.762331 12648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:25.762385 12648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:25.762426 12648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:55:25.764508 12648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.3.keytab
--principal=kudu/127.24.255.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:33115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:25.764760 12648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:25.764961 12648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:55:25.770335 12648 server_base.cc:1034] running on GCE node
W20260926 05:55:25.770345 12654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:25.770360 12656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:25.770473 12653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:25.770782 12648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:25.771024 12648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:25.772212 12648 hybrid_clock.cc:648] HybridClock initialized: now 1790402125772192 us; error 35 us; skew 500 ppm
Sep 26 05:55:25 dist-test-slave-7fr8 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402125, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:25.774987 12648 init.cc:377] Logged in from keytab as kudu/127.24.255.3@KRBTEST.COM (short username kudu)
I20260926 05:55:25.775432 12648 webserver.cc:460] Webserver started at http://127.24.255.3:39313/ using document root <none> and password file <none>
I20260926 05:55:25.775698 12648 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:25.775750 12648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:25.775871 12648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:25.776916 12648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "2ada5b22e16c4529b554582b5ba2e4e4"
format_stamp: "Formatted at 2026-09-26 05:55:25 on dist-test-slave-7fr8"
server_key: "0ff3573ad80501587f6617c9e620b7ef"
server_key_iv: "4645bedfb6856b944ba503f19bfe86bd"
server_key_version: "encryptionkey@0"
I20260926 05:55:25.777251 12648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "2ada5b22e16c4529b554582b5ba2e4e4"
format_stamp: "Formatted at 2026-09-26 05:55:25 on dist-test-slave-7fr8"
server_key: "0ff3573ad80501587f6617c9e620b7ef"
server_key_iv: "4645bedfb6856b944ba503f19bfe86bd"
server_key_version: "encryptionkey@0"
I20260926 05:55:25.778715 12648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:55:25.779656 12663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:25.779860 12648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 05:55:25.779939 12648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "2ada5b22e16c4529b554582b5ba2e4e4"
format_stamp: "Formatted at 2026-09-26 05:55:25 on dist-test-slave-7fr8"
server_key: "0ff3573ad80501587f6617c9e620b7ef"
server_key_iv: "4645bedfb6856b944ba503f19bfe86bd"
server_key_version: "encryptionkey@0"
I20260926 05:55:25.780009 12648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:25.789553 12648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:25.791812 12648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:25.791975 12648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:25.792202 12648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:25.792538 12648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:25.792572 12648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:25.792609 12648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:25.792634 12648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:25.798318 12648 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:41717
I20260926 05:55:25.798451 12776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:41717 every 8 connection(s)
I20260926 05:55:25.798748 12648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:55:25.803225 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 12648
I20260926 05:55:25.803321 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:55:25.803478 25596 external_mini_cluster.cc:1428] Setting key 25d97d10f22f2b72554c3de3cc0a9dc5
Sep 26 05:55:25 dist-test-slave-7fr8 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402125, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.3@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:25.808415 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.799695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:60561 (local address 127.24.255.62:33115)
0926 05:55:25.799752 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:25.799754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:25.800029 (+ 275us) server_negotiation.cc:407] Connection header received
0926 05:55:25.800561 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:25.800563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:25.800575 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:25.800635 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:25.801635 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.802293 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.803293 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.803545 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.803618 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.805801 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:25.805802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:25.805803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:25.805811 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:25.807115 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:25.807503 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:25.807504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:25.807504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:25.807557 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:25.807775 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:25.807776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:25.807776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:25.807891 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:25.807947 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:25.808237 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 05:55:25.808319 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:55:25.808619 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.799674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:33115 (local address 127.24.255.3:60561)
0926 05:55:25.799888 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:25.799933 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:25.800460 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:25.800711 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:25.800715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:25.801305 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:25.801503 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.801507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.802388 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.802388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.803212 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.803215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.803631 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.803632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.803753 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.804235 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:25.804243 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:25.805698 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:25.807202 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:25.807210 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:25.807212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:25.807436 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:25.807613 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:25.807614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:25.807614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:25.807720 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:25.807944 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:25.807958 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:25.808105 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 05:55:25.808210 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 05:55:25.808385 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":109,"threads_started":1}
I20260926 05:55:25.809123 12777 heartbeater.cc:347] Connected to a master server at 127.24.255.62:33115
I20260926 05:55:25.809206 12777 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:25.809394 12777 heartbeater.cc:511] Master 127.24.255.62:33115 requested a full tablet report, sending...
I20260926 05:55:25.809789 12318 ts_manager.cc:194] Registered new tserver with Master: 2ada5b22e16c4529b554582b5ba2e4e4 (127.24.255.3:41717)
I20260926 05:55:25.810207 12318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.255.3@KRBTEST.COM'} at 127.24.255.3:60561
I20260926 05:55:25.815063 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:55:25.822623 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.816440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58486 (local address 127.24.255.62:33115)
0926 05:55:25.816497 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:25.816499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:25.816508 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:55:25.816563 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:25.816565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:25.816575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:25.816660 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:25.817247 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.817847 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.818880 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.819077 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.819131 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.820270 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:25.820271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:25.820272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:25.820279 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:25.821498 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:25.821803 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:25.821804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:25.821805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:25.821859 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:25.822021 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:25.822022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:25.822022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:25.822231 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:25.822320 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:25.822440 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 05:55:25.822517 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:55:25.824556 12318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:55:25.826414 12318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:55:25.833114 12790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.829053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42216 (local address 127.24.255.2:43505)
0926 05:55:25.829401 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:25.829405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:25.829415 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:55:25.829453 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:25.829455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:25.829469 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:25.829544 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:25.829717 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.830569 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.832137 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.832704 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.832771 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.832837 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:25.832974 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 05:55:25.833020 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":80,"threads_started":1}
I20260926 05:55:25.833106 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.828928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:43505 (local address 127.0.0.1:42216)
0926 05:55:25.829238 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:25.829284 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:25.829353 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:25.829557 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:25.829561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:25.829563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:25.829638 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.829641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.830665 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.830666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.832037 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.832041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.832786 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.832787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.832874 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.832889 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 05:55:25.832952 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 05:55:25.833015 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":61,"threads_started":1}
I20260926 05:55:25.833386 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.829065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.24.255.1:41707)
0926 05:55:25.829227 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:25.829232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:25.829524 (+ 292us) server_negotiation.cc:407] Connection header received
0926 05:55:25.829658 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:25.829661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:25.829676 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:25.829725 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:25.829930 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.831030 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.832176 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.832939 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.833010 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.833067 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:25.833266 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 05:55:25.833308 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260926 05:55:25.833602 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.828929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:41717 (local address 127.0.0.1:57494)
0926 05:55:25.829334 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:25.829344 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:25.829394 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:25.829660 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:25.829661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:25.829663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:25.829719 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.829721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.831039 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.831040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.832146 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.832148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.833105 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.833105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.833172 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.833178 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:55:25.833473 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 05:55:25.833522 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":31,"thread_start_us":79,"threads_started":1}
I20260926 05:55:25.833617 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.828929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:41707 (local address 127.0.0.1:37070)
0926 05:55:25.829468 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:25.829476 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:25.829550 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:25.829822 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:25.829822 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:25.829824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:25.829860 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.829862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.831132 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.831133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.832016 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.832107 (+ 91us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.833103 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.833104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.833172 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.833179 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 05:55:25.833517 (+ 338us) client_negotiation.cc:240] Negotiation successful
0926 05:55:25.833554 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":285,"thread_start_us":68,"threads_started":1}
I20260926 05:55:25.833925 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.829192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.24.255.3:41717)
0926 05:55:25.829368 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:25.829372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:25.829385 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:55:25.829460 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:25.829462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:25.829478 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:25.829597 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:25.829787 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.830935 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.832229 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.832939 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.833016 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.833067 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:25.833790 (+ 723us) server_negotiation.cc:299] Negotiation successful
0926 05:55:25.833839 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260926 05:55:25.833959 12577 tablet_service.cc:1467] Processing CreateTablet for tablet d5ab7c6ff24646a6a9ebd7ad04101dba (DEFAULT_TABLE table=test-table [id=2164ed6b858d4c088074e1fa14bfacd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:25.834393 12577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ab7c6ff24646a6a9ebd7ad04101dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:25.834444 12443 tablet_service.cc:1467] Processing CreateTablet for tablet d5ab7c6ff24646a6a9ebd7ad04101dba (DEFAULT_TABLE table=test-table [id=2164ed6b858d4c088074e1fa14bfacd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:25.834558 12711 tablet_service.cc:1467] Processing CreateTablet for tablet d5ab7c6ff24646a6a9ebd7ad04101dba (DEFAULT_TABLE table=test-table [id=2164ed6b858d4c088074e1fa14bfacd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:25.834741 12443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ab7c6ff24646a6a9ebd7ad04101dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:25.834805 12711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ab7c6ff24646a6a9ebd7ad04101dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:25.837419 12794 tablet_bootstrap.cc:492] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4: Bootstrap starting.
I20260926 05:55:25.837747 12795 tablet_bootstrap.cc:492] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779: Bootstrap starting.
I20260926 05:55:25.838195 12794 tablet_bootstrap.cc:654] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:25.838264 12796 tablet_bootstrap.cc:492] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6: Bootstrap starting.
I20260926 05:55:25.838351 12795 tablet_bootstrap.cc:654] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:25.838686 12794 log.cc:824] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:25.838878 12796 tablet_bootstrap.cc:654] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:25.838865 12795 log.cc:824] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:25.839327 12796 log.cc:824] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:25.839370 12794 tablet_bootstrap.cc:492] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4: No bootstrap required, opened a new log
I20260926 05:55:25.839414 12794 ts_tablet_manager.cc:1397] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:55:25.840083 12796 tablet_bootstrap.cc:492] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6: No bootstrap required, opened a new log
I20260926 05:55:25.840162 12796 ts_tablet_manager.cc:1397] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:55:25.841184 12794 raft_consensus.cc:348] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ada5b22e16c4529b554582b5ba2e4e4" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 41717 } } peers { permanent_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41707 } } peers { permanent_uuid: "d3b36afb8bcb4c278f915aff2b493779" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43505 } }
I20260926 05:55:25.841331 12794 raft_consensus.cc:374] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:25.841351 12794 raft_consensus.cc:729] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ada5b22e16c4529b554582b5ba2e4e4, State: Initialized, Role: FOLLOWER
I20260926 05:55:25.841456 12794 consensus_queue.cc:260] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ada5b22e16c4529b554582b5ba2e4e4" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 41717 } } peers { permanent_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41707 } } peers { permanent_uuid: "d3b36afb8bcb4c278f915aff2b493779" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43505 } }
I20260926 05:55:25.841714 12794 ts_tablet_manager.cc:1428] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 05:55:25.841755 12777 heartbeater.cc:503] Master 127.24.255.62:33115 was elected leader, sending a full tablet report...
I20260926 05:55:25.842356 12795 tablet_bootstrap.cc:492] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779: No bootstrap required, opened a new log
I20260926 05:55:25.842211 12796 raft_consensus.cc:348] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ada5b22e16c4529b554582b5ba2e4e4" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 41717 } } peers { permanent_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41707 } } peers { permanent_uuid: "d3b36afb8bcb4c278f915aff2b493779" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43505 } }
I20260926 05:55:25.842403 12796 raft_consensus.cc:374] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:25.842425 12795 ts_tablet_manager.cc:1397] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 05:55:25.842442 12796 raft_consensus.cc:729] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0c8c3e221b94a5d9f448d5c5a6ae6c6, State: Initialized, Role: FOLLOWER
I20260926 05:55:25.842584 12796 consensus_queue.cc:260] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ada5b22e16c4529b554582b5ba2e4e4" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 41717 } } peers { permanent_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41707 } } peers { permanent_uuid: "d3b36afb8bcb4c278f915aff2b493779" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43505 } }
I20260926 05:55:25.842905 12796 ts_tablet_manager.cc:1428] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:55:25.842947 12509 heartbeater.cc:503] Master 127.24.255.62:33115 was elected leader, sending a full tablet report...
I20260926 05:55:25.844121 12795 raft_consensus.cc:348] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ada5b22e16c4529b554582b5ba2e4e4" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 41717 } } peers { permanent_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41707 } } peers { permanent_uuid: "d3b36afb8bcb4c278f915aff2b493779" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43505 } }
I20260926 05:55:25.844259 12795 raft_consensus.cc:374] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:25.844290 12795 raft_consensus.cc:729] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3b36afb8bcb4c278f915aff2b493779, State: Initialized, Role: FOLLOWER
I20260926 05:55:25.844403 12795 consensus_queue.cc:260] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ada5b22e16c4529b554582b5ba2e4e4" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 41717 } } peers { permanent_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41707 } } peers { permanent_uuid: "d3b36afb8bcb4c278f915aff2b493779" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43505 } }
I20260926 05:55:25.844655 12795 ts_tablet_manager.cc:1428] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260926 05:55:25.844719 12643 heartbeater.cc:503] Master 127.24.255.62:33115 was elected leader, sending a full tablet report...
I20260926 05:55:25.847236 12801 raft_consensus.cc:482] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:55:25.847361 12801 raft_consensus.cc:504] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ada5b22e16c4529b554582b5ba2e4e4" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 41717 } } peers { permanent_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41707 } } peers { permanent_uuid: "d3b36afb8bcb4c278f915aff2b493779" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43505 } }
I20260926 05:55:25.847695 12801 leader_election.cc:288] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ada5b22e16c4529b554582b5ba2e4e4 (127.24.255.3:41717), d3b36afb8bcb4c278f915aff2b493779 (127.24.255.2:43505)
I20260926 05:55:25.851141 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.847897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:43505 (local address 127.24.255.1:40423)
0926 05:55:25.847953 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:25.847965 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:25.848004 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:25.848247 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:25.848249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:25.848251 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:25.848323 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.848325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.849112 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.849113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.850155 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.850157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.850892 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.850892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.850973 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.850976 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:55:25.851019 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 05:55:25.851064 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:55:25.851141 12790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.847938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:40423 (local address 127.24.255.2:43505)
0926 05:55:25.848150 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:25.848151 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:25.848158 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:55:25.848177 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:25.848178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:25.848185 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:25.848238 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:25.848384 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.849013 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.850252 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.850826 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.850889 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.850899 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:25.851039 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 05:55:25.851076 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260926 05:55:25.851444 12597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ab7c6ff24646a6a9ebd7ad04101dba" candidate_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3b36afb8bcb4c278f915aff2b493779" is_pre_election: true
I20260926 05:55:25.851593 12597 raft_consensus.cc:2454] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0c8c3e221b94a5d9f448d5c5a6ae6c6 in term 0.
I20260926 05:55:25.851713 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.847897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:41717 (local address 127.24.255.1:44637)
0926 05:55:25.848247 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:25.848270 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:25.848397 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:25.848626 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:25.848627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:25.848628 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:25.848771 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.848776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.849552 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.849553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.850680 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:25.850683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.851325 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.851326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:25.851489 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.851495 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:55:25.851582 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 05:55:25.851629 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":60,"thread_start_us":102,"threads_started":1}
I20260926 05:55:25.851735 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.848085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:44637 (local address 127.24.255.3:41717)
0926 05:55:25.848291 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:25.848294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:25.848367 (+ 73us) server_negotiation.cc:407] Connection header received
0926 05:55:25.848532 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:25.848535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:25.848545 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:25.848610 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:25.848863 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.849433 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.850769 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.851174 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.851270 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.851281 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:25.851627 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 05:55:25.851677 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260926 05:55:25.852030 12399 leader_election.cc:302] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0c8c3e221b94a5d9f448d5c5a6ae6c6, d3b36afb8bcb4c278f915aff2b493779; no voters:
I20260926 05:55:25.852106 12731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ab7c6ff24646a6a9ebd7ad04101dba" candidate_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ada5b22e16c4529b554582b5ba2e4e4" is_pre_election: true
I20260926 05:55:25.852243 12731 raft_consensus.cc:2454] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0c8c3e221b94a5d9f448d5c5a6ae6c6 in term 0.
I20260926 05:55:25.852212 12801 raft_consensus.cc:2781] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:55:25.852284 12801 raft_consensus.cc:482] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:55:25.852314 12801 raft_consensus.cc:3037] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:25.853125 12801 raft_consensus.cc:504] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ada5b22e16c4529b554582b5ba2e4e4" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 41717 } } peers { permanent_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41707 } } peers { permanent_uuid: "d3b36afb8bcb4c278f915aff2b493779" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43505 } }
I20260926 05:55:25.853255 12801 leader_election.cc:288] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [CANDIDATE]: Term 1 election: Requested vote from peers 2ada5b22e16c4529b554582b5ba2e4e4 (127.24.255.3:41717), d3b36afb8bcb4c278f915aff2b493779 (127.24.255.2:43505)
I20260926 05:55:25.853415 12731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ab7c6ff24646a6a9ebd7ad04101dba" candidate_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ada5b22e16c4529b554582b5ba2e4e4"
I20260926 05:55:25.853406 12597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ab7c6ff24646a6a9ebd7ad04101dba" candidate_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3b36afb8bcb4c278f915aff2b493779"
I20260926 05:55:25.853494 12597 raft_consensus.cc:3037] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:25.853502 12731 raft_consensus.cc:3037] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:25.854388 12731 raft_consensus.cc:2454] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0c8c3e221b94a5d9f448d5c5a6ae6c6 in term 1.
I20260926 05:55:25.854389 12597 raft_consensus.cc:2454] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0c8c3e221b94a5d9f448d5c5a6ae6c6 in term 1.
I20260926 05:55:25.854578 12399 leader_election.cc:302] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0c8c3e221b94a5d9f448d5c5a6ae6c6, d3b36afb8bcb4c278f915aff2b493779; no voters:
I20260926 05:55:25.854673 12801 raft_consensus.cc:2781] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:25.854813 12801 raft_consensus.cc:686] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [term 1 LEADER]: Becoming Leader. State: Replica: d0c8c3e221b94a5d9f448d5c5a6ae6c6, State: Running, Role: LEADER
I20260926 05:55:25.854892 12801 consensus_queue.cc:237] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ada5b22e16c4529b554582b5ba2e4e4" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 41717 } } peers { permanent_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41707 } } peers { permanent_uuid: "d3b36afb8bcb4c278f915aff2b493779" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43505 } }
I20260926 05:55:25.855798 12318 catalog_manager.cc:5495] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0c8c3e221b94a5d9f448d5c5a6ae6c6 (127.24.255.1). New cstate: current_term: 1 leader_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ada5b22e16c4529b554582b5ba2e4e4" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0c8c3e221b94a5d9f448d5c5a6ae6c6" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 41707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3b36afb8bcb4c278f915aff2b493779" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43505 } health_report { overall_health: UNKNOWN } } }
I20260926 05:55:25.861126 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.858745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.24.255.1:41707)
0926 05:55:25.858818 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:25.858820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:25.858860 (+ 40us) server_negotiation.cc:407] Connection header received
0926 05:55:25.858959 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:25.858960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:25.858979 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:25.859048 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:25.859179 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.859655 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.860388 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:25.860573 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:25.860627 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:25.860776 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:25.860894 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:25.860964 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:25.861010 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 05:55:25.861030 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 05:55:25.863197 12731 raft_consensus.cc:1261] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4 [term 1 FOLLOWER]: Refusing update from remote peer d0c8c3e221b94a5d9f448d5c5a6ae6c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:25.863224 12597 raft_consensus.cc:1261] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779 [term 1 FOLLOWER]: Refusing update from remote peer d0c8c3e221b94a5d9f448d5c5a6ae6c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:25.863590 12801 consensus_queue.cc:1035] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ada5b22e16c4529b554582b5ba2e4e4" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:25.863762 12804 consensus_queue.cc:1035] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3b36afb8bcb4c278f915aff2b493779" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:25.868790 12318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:55:25.868927 12318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:55:25.869724 12318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce2ebeccd6cb42d788eff4525f53fe50: Sending DeleteTablet for 3 replicas of tablet d5ab7c6ff24646a6a9ebd7ad04101dba
I20260926 05:55:25.870134 12711 tablet_service.cc:1514] Processing DeleteTablet for tablet d5ab7c6ff24646a6a9ebd7ad04101dba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:25 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:57494
I20260926 05:55:25.870134 12577 tablet_service.cc:1514] Processing DeleteTablet for tablet d5ab7c6ff24646a6a9ebd7ad04101dba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:25 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:42216
I20260926 05:55:25.870232 12443 tablet_service.cc:1514] Processing DeleteTablet for tablet d5ab7c6ff24646a6a9ebd7ad04101dba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:25 UTC) from {username='kudu', principal='kudu/127.24.255.62@KRBTEST.COM'} at 127.0.0.1:37070
I20260926 05:55:25.870561 12818 tablet_replica.cc:330] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6: stopping tablet replica
I20260926 05:55:25.870668 12818 raft_consensus.cc:2229] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:55:25.870762 12818 raft_consensus.cc:2258] T d5ab7c6ff24646a6a9ebd7ad04101dba P d0c8c3e221b94a5d9f448d5c5a6ae6c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:25.870853 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 12380
I20260926 05:55:25.871562 12817 tablet_replica.cc:330] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779: stopping tablet replica
I20260926 05:55:25.871680 12817 raft_consensus.cc:2229] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:55:25.871776 12817 raft_consensus.cc:2258] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:25.872052 12816 tablet_replica.cc:330] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4: stopping tablet replica
I20260926 05:55:25.872169 12816 raft_consensus.cc:2229] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:55:25.872233 12817 ts_tablet_manager.cc:1905] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:55:25.872258 12816 raft_consensus.cc:2258] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:25.872733 12816 ts_tablet_manager.cc:1905] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:55:25.874198 12817 ts_tablet_manager.cc:1918] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:55:25.874233 12816 ts_tablet_manager.cc:1918] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:55:25.874290 12816 log.cc:1196] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/d5ab7c6ff24646a6a9ebd7ad04101dba
I20260926 05:55:25.874286 12817 log.cc:1196] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/d5ab7c6ff24646a6a9ebd7ad04101dba
I20260926 05:55:25.874533 12816 ts_tablet_manager.cc:1939] T d5ab7c6ff24646a6a9ebd7ad04101dba P 2ada5b22e16c4529b554582b5ba2e4e4: Deleting consensus metadata
I20260926 05:55:25.874589 12817 ts_tablet_manager.cc:1939] T d5ab7c6ff24646a6a9ebd7ad04101dba P d3b36afb8bcb4c278f915aff2b493779: Deleting consensus metadata
I20260926 05:55:25.875092 12305 catalog_manager.cc:4841] TS d3b36afb8bcb4c278f915aff2b493779 (127.24.255.2:43505): tablet d5ab7c6ff24646a6a9ebd7ad04101dba (table test-table [id=2164ed6b858d4c088074e1fa14bfacd2]) successfully deleted
I20260926 05:55:25.875092 12304 catalog_manager.cc:4841] TS 2ada5b22e16c4529b554582b5ba2e4e4 (127.24.255.3:41717): tablet d5ab7c6ff24646a6a9ebd7ad04101dba (table test-table [id=2164ed6b858d4c088074e1fa14bfacd2]) successfully deleted
W20260926 05:55:25.876904 12302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:41707 (error 108)
I20260926 05:55:25.877087 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 12514
I20260926 05:55:25.878304 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:25.877712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:41707 (local address 127.0.0.1:37090)
0926 05:55:25.878227 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:25.878270 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:41707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488}
W20260926 05:55:25.878427 12302 catalog_manager.cc:4584] TS d0c8c3e221b94a5d9f448d5c5a6ae6c6 (127.24.255.1:41707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5ab7c6ff24646a6a9ebd7ad04101dba: Network error: Client connection negotiation failed: client connection to 127.24.255.1:41707: connect: Connection refused (error 111)
I20260926 05:55:25.881341 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 12648
I20260926 05:55:25.886622 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 12286
2026-09-26T05:55:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1825 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T05:55:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:25Z Disabled control of system clock
I20260926 05:55:25.915037 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:38527
--webserver_interface=127.24.255.62
--webserver_port=0
--builtin_ntp_servers=127.24.255.20:36687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:38527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:55:26.001492 12823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:26.001765 12823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:26.001788 12823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:26.003077 12823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:26.003137 12823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:26.003149 12823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:26.003160 12823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:26.003171 12823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:26.004606 12823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:38527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:38527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:26.004839 12823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:26.005110 12823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:55:26.010468 12823 server_base.cc:1034] running on GCE node
W20260926 05:55:26.010463 12828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:26.010435 12829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:26.010614 12831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:26.010955 12823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:26.011264 12823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:26.012476 12823 hybrid_clock.cc:648] HybridClock initialized: now 1790402126012416 us; error 63 us; skew 500 ppm
I20260926 05:55:26.014619 12823 webserver.cc:460] Webserver started at http://127.24.255.62:43577/ using document root <none> and password file <none>
I20260926 05:55:26.014899 12823 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:26.014956 12823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:26.015089 12823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:26.016259 12823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "49fe0a1b812948c988b96f2408e0f870"
format_stamp: "Formatted at 2026-09-26 05:55:26 on dist-test-slave-7fr8"
server_key: "c4317ee9de314b184982fa84026c34e0"
server_key_iv: "f68b28a04f712718ba47903aef21dede"
server_key_version: "encryptionkey@0"
I20260926 05:55:26.016618 12823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "49fe0a1b812948c988b96f2408e0f870"
format_stamp: "Formatted at 2026-09-26 05:55:26 on dist-test-slave-7fr8"
server_key: "c4317ee9de314b184982fa84026c34e0"
server_key_iv: "f68b28a04f712718ba47903aef21dede"
server_key_version: "encryptionkey@0"
I20260926 05:55:26.018042 12823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 05:55:26.019127 12837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:26.019622 12823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:26.019761 12823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "49fe0a1b812948c988b96f2408e0f870"
format_stamp: "Formatted at 2026-09-26 05:55:26 on dist-test-slave-7fr8"
server_key: "c4317ee9de314b184982fa84026c34e0"
server_key_iv: "f68b28a04f712718ba47903aef21dede"
server_key_version: "encryptionkey@0"
I20260926 05:55:26.019840 12823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:26.033201 12823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:26.033558 12823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:26.033690 12823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:26.037987 12823 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:38527
I20260926 05:55:26.038031 12889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:38527 every 8 connection(s)
I20260926 05:55:26.038422 12823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:26.039408 12890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:26.040593 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 12823
I20260926 05:55:26.040709 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:26.040871 25596 external_mini_cluster.cc:1428] Setting key ee1b54c3f41b613263a8d0ae28461eca
I20260926 05:55:26.042366 12890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870: Bootstrap starting.
I20260926 05:55:26.043123 12890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:26.043447 12890 log.cc:824] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:26.044245 12890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870: No bootstrap required, opened a new log
I20260926 05:55:26.046542 12890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fe0a1b812948c988b96f2408e0f870" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38527 } }
I20260926 05:55:26.046744 12890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:26.046784 12890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fe0a1b812948c988b96f2408e0f870, State: Initialized, Role: FOLLOWER
I20260926 05:55:26.046969 12890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fe0a1b812948c988b96f2408e0f870" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38527 } }
I20260926 05:55:26.047052 12890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:26.047122 12890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:26.047180 12890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:26.047334 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.041482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.24.255.62:38527)
0926 05:55:26.041701 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:26.041710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:26.041744 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:55:26.042303 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:26.042311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:26.042392 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:26.042553 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:26.042767 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.044009 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.044832 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.045467 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.045597 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.046426 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:26.046430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:26.046430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:26.046636 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:26.046680 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:26.046695 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:26.046763 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:26.046891 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 05:55:26.047027 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260926 05:55:26.047925 12890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fe0a1b812948c988b96f2408e0f870" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38527 } }
I20260926 05:55:26.048039 12890 leader_election.cc:302] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49fe0a1b812948c988b96f2408e0f870; no voters:
I20260926 05:55:26.048238 12890 leader_election.cc:288] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:26.048282 12895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:26.048444 12895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [term 1 LEADER]: Becoming Leader. State: Replica: 49fe0a1b812948c988b96f2408e0f870, State: Running, Role: LEADER
I20260926 05:55:26.048501 12890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:26.048543 12895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fe0a1b812948c988b96f2408e0f870" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38527 } }
I20260926 05:55:26.048934 12896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49fe0a1b812948c988b96f2408e0f870" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fe0a1b812948c988b96f2408e0f870" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38527 } } }
I20260926 05:55:26.048955 12897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49fe0a1b812948c988b96f2408e0f870. Latest consensus state: current_term: 1 leader_uuid: "49fe0a1b812948c988b96f2408e0f870" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fe0a1b812948c988b96f2408e0f870" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 38527 } } }
I20260926 05:55:26.049052 12897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:26.049042 12896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:26.049268 12904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:26.049754 12904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:26.051445 12904 catalog_manager.cc:1347] Generated new cluster ID: 70a622ba77274136b04c2ea27a465fbf
I20260926 05:55:26.051507 12904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:26.066928 12904 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:26.067703 12904 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:26.084684 12904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870: Generated new TSK 0
I20260926 05:55:26.084945 12904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:26.086448 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.1:0
--local_ip_for_outbound_sockets=127.24.255.1
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38527
--builtin_ntp_servers=127.24.255.20:36687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:55:26.171778 12914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:26.171993 12914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:26.172016 12914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:26.173257 12914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:26.173302 12914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:26.173329 12914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.1
I20260926 05:55:26.174777 12914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.255.1
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.1
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:26.174988 12914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:26.175298 12914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:26.180411 12922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:26.180428 12919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:26.180655 12914 server_base.cc:1034] running on GCE node
W20260926 05:55:26.180431 12920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:26.181020 12914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:26.181305 12914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:26.182461 12914 hybrid_clock.cc:648] HybridClock initialized: now 1790402126182442 us; error 34 us; skew 500 ppm
I20260926 05:55:26.184093 12914 webserver.cc:460] Webserver started at http://127.24.255.1:37841/ using document root <none> and password file <none>
I20260926 05:55:26.184360 12914 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:26.184418 12914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:26.184546 12914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:26.186208 12914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/data/instance:
uuid: "896c38a41da64789aa78026e5e19770f"
format_stamp: "Formatted at 2026-09-26 05:55:26 on dist-test-slave-7fr8"
server_key: "c37414be807e8d02020e68c63b022a2b"
server_key_iv: "a56943ddeae832d9cbb5b05f64b7eb80"
server_key_version: "encryptionkey@0"
I20260926 05:55:26.186638 12914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance:
uuid: "896c38a41da64789aa78026e5e19770f"
format_stamp: "Formatted at 2026-09-26 05:55:26 on dist-test-slave-7fr8"
server_key: "c37414be807e8d02020e68c63b022a2b"
server_key_iv: "a56943ddeae832d9cbb5b05f64b7eb80"
server_key_version: "encryptionkey@0"
I20260926 05:55:26.188612 12914 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260926 05:55:26.189810 12928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:26.190176 12914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:26.190294 12914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/wal
uuid: "896c38a41da64789aa78026e5e19770f"
format_stamp: "Formatted at 2026-09-26 05:55:26 on dist-test-slave-7fr8"
server_key: "c37414be807e8d02020e68c63b022a2b"
server_key_iv: "a56943ddeae832d9cbb5b05f64b7eb80"
server_key_version: "encryptionkey@0"
I20260926 05:55:26.190387 12914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:26.209198 12914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:26.209506 12914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:26.209612 12914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:26.209820 12914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:26.210234 12914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:26.210274 12914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:26.210297 12914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:26.210309 12914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:26.216110 12914 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.1:39555
I20260926 05:55:26.216171 13041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.1:39555 every 8 connection(s)
I20260926 05:55:26.216503 12914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/data/info.pb
I20260926 05:55:26.221666 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 12914
I20260926 05:55:26.221789 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-0/wal/instance
I20260926 05:55:26.221979 25596 external_mini_cluster.cc:1428] Setting key e95e3e94aa54a728282442ec11280001
I20260926 05:55:26.222587 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.217254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.1:49731 (local address 127.24.255.62:38527)
0926 05:55:26.217317 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:26.217320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:26.217669 (+ 349us) server_negotiation.cc:407] Connection header received
0926 05:55:26.218203 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:26.218205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:26.218216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:26.218267 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:26.218910 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.219417 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.220525 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.220745 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.220810 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.222312 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:26.222314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:26.222314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:26.222319 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:26.222337 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:26.222342 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:26.222386 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:26.222482 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 05:55:26.222510 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:55:26.222872 13044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.217246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38527 (local address 127.24.255.1:49731)
0926 05:55:26.217495 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:26.217534 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:26.218046 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:26.218328 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:26.218332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:26.218340 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:26.218844 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.218848 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.219590 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.219591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.220406 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.220409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.220932 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.220935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.221156 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.222129 (+ 973us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:55:26.222130 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:26.222176 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:55:26.222207 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:26.222399 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:26.222403 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:26.222406 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:55:26.222475 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 05:55:26.222634 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260926 05:55:26.223099 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.2:0
--local_ip_for_outbound_sockets=127.24.255.2
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38527
--builtin_ntp_servers=127.24.255.20:36687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 05:55:26.223440 13042 heartbeater.cc:347] Connected to a master server at 127.24.255.62:38527
I20260926 05:55:26.223548 13042 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:26.223791 13042 heartbeater.cc:511] Master 127.24.255.62:38527 requested a full tablet report, sending...
I20260926 05:55:26.224346 12854 ts_manager.cc:194] Registered new tserver with Master: 896c38a41da64789aa78026e5e19770f (127.24.255.1:39555)
I20260926 05:55:26.225042 12854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.255.1:49731
W20260926 05:55:26.312508 13045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:26.312739 13045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:26.312770 13045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:26.314078 13045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:26.314175 13045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:26.314208 13045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.2
I20260926 05:55:26.315666 13045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.255.2
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.2
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:26.315888 13045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:26.316154 13045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:26.321178 13053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:26.321184 13050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:26.321169 13051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:26.321609 13045 server_base.cc:1034] running on GCE node
I20260926 05:55:26.321818 13045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:26.322049 13045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:26.323280 13045 hybrid_clock.cc:648] HybridClock initialized: now 1790402126323265 us; error 30 us; skew 500 ppm
I20260926 05:55:26.324640 13045 webserver.cc:460] Webserver started at http://127.24.255.2:43759/ using document root <none> and password file <none>
I20260926 05:55:26.324885 13045 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:26.324937 13045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:26.325075 13045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:26.326200 13045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/data/instance:
uuid: "1d57f5a9d7084952badee6ab559f5b3c"
format_stamp: "Formatted at 2026-09-26 05:55:26 on dist-test-slave-7fr8"
server_key: "88ff42670ecc98dcabd9d3db9726bb15"
server_key_iv: "e8fb5fd4a63e476ec89484686a34dfbb"
server_key_version: "encryptionkey@0"
I20260926 05:55:26.326553 13045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance:
uuid: "1d57f5a9d7084952badee6ab559f5b3c"
format_stamp: "Formatted at 2026-09-26 05:55:26 on dist-test-slave-7fr8"
server_key: "88ff42670ecc98dcabd9d3db9726bb15"
server_key_iv: "e8fb5fd4a63e476ec89484686a34dfbb"
server_key_version: "encryptionkey@0"
I20260926 05:55:26.327865 13045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:55:26.328909 13059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:26.329064 13045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:55:26.329151 13045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/wal
uuid: "1d57f5a9d7084952badee6ab559f5b3c"
format_stamp: "Formatted at 2026-09-26 05:55:26 on dist-test-slave-7fr8"
server_key: "88ff42670ecc98dcabd9d3db9726bb15"
server_key_iv: "e8fb5fd4a63e476ec89484686a34dfbb"
server_key_version: "encryptionkey@0"
I20260926 05:55:26.329221 13045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:26.343685 13045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:26.344025 13045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:26.344158 13045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:26.344405 13045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:26.344795 13045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:26.344830 13045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:26.344861 13045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:26.344877 13045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:26.351135 13045 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.2:41987
I20260926 05:55:26.351226 13172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.2:41987 every 8 connection(s)
I20260926 05:55:26.351589 13045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/data/info.pb
I20260926 05:55:26.358067 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.352636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:50659 (local address 127.24.255.62:38527)
0926 05:55:26.352759 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:26.352762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:26.352977 (+ 215us) server_negotiation.cc:407] Connection header received
0926 05:55:26.353515 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:26.353517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:26.353531 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:26.353598 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:26.354650 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.355150 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.356262 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.356457 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.356531 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.357623 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:26.357626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:26.357626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:26.357635 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:26.357663 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:26.357669 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:26.357761 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:26.357950 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 05:55:26.357982 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 05:55:26.358299 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.352519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38527 (local address 127.24.255.2:50659)
0926 05:55:26.352803 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:26.352848 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:26.353397 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:26.353772 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:26.353778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:26.353791 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:26.354510 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.354519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.355269 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.355271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.356158 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.356160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.356632 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.356634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.356803 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.357399 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:55:26.357399 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:26.357430 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:55:26.357458 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:26.357802 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:26.357809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:26.357812 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:55:26.357902 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 05:55:26.358005 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":141,"threads_started":1}
I20260926 05:55:26.358907 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 13045
I20260926 05:55:26.358999 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/wal/instance
I20260926 05:55:26.359102 13173 heartbeater.cc:347] Connected to a master server at 127.24.255.62:38527
I20260926 05:55:26.359195 25596 external_mini_cluster.cc:1428] Setting key a2d5684d24e6b2f681f3f9f1bd0c913f
I20260926 05:55:26.359216 13173 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:26.359521 13173 heartbeater.cc:511] Master 127.24.255.62:38527 requested a full tablet report, sending...
I20260926 05:55:26.359979 12854 ts_manager.cc:194] Registered new tserver with Master: 1d57f5a9d7084952badee6ab559f5b3c (127.24.255.2:41987)
I20260926 05:55:26.360363 12854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.255.2:50659
I20260926 05:55:26.360561 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.255.3:0
--local_ip_for_outbound_sockets=127.24.255.3
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38527
--builtin_ntp_servers=127.24.255.20:36687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:55:26.451402 13176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:26.451649 13176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:26.451684 13176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:26.452955 13176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:26.453003 13176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:26.453040 13176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.255.3
I20260926 05:55:26.454542 13176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.255.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.255.3
--webserver_port=0
--tserver_master_addrs=127.24.255.62:38527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.255.3
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:26.454811 13176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:26.455111 13176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:26.460232 13181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:26.460405 13176 server_base.cc:1034] running on GCE node
W20260926 05:55:26.460232 13184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:26.460284 13182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:26.460753 13176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:26.461001 13176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:26.462194 13176 hybrid_clock.cc:648] HybridClock initialized: now 1790402126462171 us; error 56 us; skew 500 ppm
I20260926 05:55:26.463464 13176 webserver.cc:460] Webserver started at http://127.24.255.3:33395/ using document root <none> and password file <none>
I20260926 05:55:26.463690 13176 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:26.463740 13176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:26.463868 13176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:26.464938 13176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/data/instance:
uuid: "b7207a9da97c4522a959146d91a35095"
format_stamp: "Formatted at 2026-09-26 05:55:26 on dist-test-slave-7fr8"
server_key: "b12b56cda27eda59d224580ef54136cc"
server_key_iv: "5eab2528b44bdf718e3fe8b898d7c673"
server_key_version: "encryptionkey@0"
I20260926 05:55:26.465281 13176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance:
uuid: "b7207a9da97c4522a959146d91a35095"
format_stamp: "Formatted at 2026-09-26 05:55:26 on dist-test-slave-7fr8"
server_key: "b12b56cda27eda59d224580ef54136cc"
server_key_iv: "5eab2528b44bdf718e3fe8b898d7c673"
server_key_version: "encryptionkey@0"
I20260926 05:55:26.466753 13176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:55:26.467561 13190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:26.467702 13176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:55:26.467788 13176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/wal
uuid: "b7207a9da97c4522a959146d91a35095"
format_stamp: "Formatted at 2026-09-26 05:55:26 on dist-test-slave-7fr8"
server_key: "b12b56cda27eda59d224580ef54136cc"
server_key_iv: "5eab2528b44bdf718e3fe8b898d7c673"
server_key_version: "encryptionkey@0"
I20260926 05:55:26.467854 13176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:26.477494 13176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:26.477801 13176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:26.477928 13176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:26.478188 13176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:55:26.478507 13176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:55:26.478540 13176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:26.478569 13176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:55:26.478585 13176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:26.485101 13176 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.3:35393
I20260926 05:55:26.485155 13303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.3:35393 every 8 connection(s)
I20260926 05:55:26.485515 13176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/data/info.pb
I20260926 05:55:26.486640 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 13176
I20260926 05:55:26.486815 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/wal/instance
I20260926 05:55:26.487087 25596 external_mini_cluster.cc:1428] Setting key 9b017ce78854f073f80e7224df6b1ce6
I20260926 05:55:26.492959 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.486644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.3:48867 (local address 127.24.255.62:38527)
0926 05:55:26.486749 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:26.486751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:26.487232 (+ 481us) server_negotiation.cc:407] Connection header received
0926 05:55:26.488276 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:26.488278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:26.488291 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:26.488362 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:26.489220 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.489742 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.490887 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.491086 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.491152 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.492536 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:26.492537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:26.492538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:26.492543 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:26.492568 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:26.492573 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:26.492622 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:26.492840 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 05:55:26.492879 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 05:55:26.493420 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.486602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:38527 (local address 127.24.255.3:48867)
0926 05:55:26.486964 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:26.487040 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:26.488099 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:26.488453 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:26.488458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:26.488471 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:26.489104 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.489112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.489854 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.489855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.490795 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.490800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.491344 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.491346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.491576 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.492313 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:55:26.492314 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:26.492359 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:55:26.492429 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:26.492681 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:26.492689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:26.492693 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:26.492807 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 05:55:26.492953 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
I20260926 05:55:26.494297 13304 heartbeater.cc:347] Connected to a master server at 127.24.255.62:38527
I20260926 05:55:26.494410 13304 heartbeater.cc:464] Registering TS with master...
I20260926 05:55:26.494606 13304 heartbeater.cc:511] Master 127.24.255.62:38527 requested a full tablet report, sending...
I20260926 05:55:26.495121 12854 ts_manager.cc:194] Registered new tserver with Master: b7207a9da97c4522a959146d91a35095 (127.24.255.3:35393)
I20260926 05:55:26.495572 12854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.255.3:48867
I20260926 05:55:26.499951 25596 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:55:26.505426 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.501783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.24.255.62:38527)
0926 05:55:26.501893 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:26.501896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:26.502090 (+ 194us) server_negotiation.cc:407] Connection header received
0926 05:55:26.502180 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:26.502181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:26.502194 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:26.502255 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:26.502492 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.503003 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.503880 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.504069 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.504147 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.505010 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:26.505012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:26.505013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:26.505019 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:26.505041 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:26.505046 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:26.505110 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:26.505280 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 05:55:26.505324 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 05:55:26.507342 12854 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:55:26.508978 12854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:55:26.515831 13314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.511577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:39555 (local address 127.0.0.1:56986)
0926 05:55:26.511760 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:26.511801 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:26.511871 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:26.512358 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:26.512361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:26.512364 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:26.512450 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.512453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.513499 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.513501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.514616 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.514619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.515510 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.515511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.515594 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.515611 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 05:55:26.515686 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 05:55:26.515740 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
I20260926 05:55:26.515843 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.511822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56986 (local address 127.24.255.1:39555)
0926 05:55:26.512186 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:26.512190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:26.512200 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:55:26.512236 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:26.512238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:26.512251 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:26.512339 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:26.512550 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.513414 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.514777 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.515416 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.515506 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.515605 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:26.515701 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 05:55:26.515742 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":80,"threads_started":1}
I20260926 05:55:26.516332 13313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.511574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.2:41987 (local address 127.0.0.1:44096)
0926 05:55:26.511760 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:26.511801 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:26.511871 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:26.512587 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:26.512588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:26.512592 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:26.512681 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.512683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.514191 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.514193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.515190 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.515193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.516079 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.516081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.516146 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.516150 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:26.516212 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 05:55:26.516255 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":64,"threads_started":1}
I20260926 05:55:26.516346 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.511989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.24.255.2:41987)
0926 05:55:26.512350 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:26.512357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:26.512376 (+ 19us) server_negotiation.cc:407] Connection header received
0926 05:55:26.512434 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:26.512436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:26.512456 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:26.512874 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:26.512943 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.514067 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.515279 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.515985 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.516066 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.516153 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:26.516227 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 05:55:26.516266 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
I20260926 05:55:26.517185 12976 tablet_service.cc:1467] Processing CreateTablet for tablet df4718eb1793436cae9a5d9b745c50ea (DEFAULT_TABLE table=test-table [id=3fde6c410acb477f8ca907c9bc238e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:26.517611 12976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4718eb1793436cae9a5d9b745c50ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:26.517750 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.511609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:35393 (local address 127.0.0.1:55718)
0926 05:55:26.512444 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:26.512455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:26.512506 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:26.512789 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:26.512791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:26.512794 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:26.512852 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.512854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.515268 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.515269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.516326 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.516329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.517470 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.517472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.517563 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.517569 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:55:26.517632 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 05:55:26.517678 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":73,"threads_started":1}
I20260926 05:55:26.517776 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.511800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.24.255.3:35393)
0926 05:55:26.512021 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:26.512027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:26.512575 (+ 548us) server_negotiation.cc:407] Connection header received
0926 05:55:26.512624 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:26.512626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:26.512642 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:26.512724 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:26.512925 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.514064 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.516516 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.517352 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.517433 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.517526 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:26.517653 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 05:55:26.517697 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":101,"threads_started":1}
I20260926 05:55:26.518512 13238 tablet_service.cc:1467] Processing CreateTablet for tablet df4718eb1793436cae9a5d9b745c50ea (DEFAULT_TABLE table=test-table [id=3fde6c410acb477f8ca907c9bc238e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:26.518863 13238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4718eb1793436cae9a5d9b745c50ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:26.518925 13107 tablet_service.cc:1467] Processing CreateTablet for tablet df4718eb1793436cae9a5d9b745c50ea (DEFAULT_TABLE table=test-table [id=3fde6c410acb477f8ca907c9bc238e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:55:26.519256 13107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4718eb1793436cae9a5d9b745c50ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:26.520990 13318 tablet_bootstrap.cc:492] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f: Bootstrap starting.
I20260926 05:55:26.521787 13318 tablet_bootstrap.cc:654] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:26.522054 13318 log.cc:824] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:26.522383 13319 tablet_bootstrap.cc:492] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095: Bootstrap starting.
I20260926 05:55:26.522488 13320 tablet_bootstrap.cc:492] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c: Bootstrap starting.
I20260926 05:55:26.522967 13318 tablet_bootstrap.cc:492] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f: No bootstrap required, opened a new log
I20260926 05:55:26.523020 13320 tablet_bootstrap.cc:654] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:26.523022 13319 tablet_bootstrap.cc:654] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:26.523042 13318 ts_tablet_manager.cc:1397] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:55:26.523298 13320 log.cc:824] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:26.523315 13319 log.cc:824] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:26.524017 13319 tablet_bootstrap.cc:492] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095: No bootstrap required, opened a new log
I20260926 05:55:26.524096 13319 ts_tablet_manager.cc:1397] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:55:26.524248 13320 tablet_bootstrap.cc:492] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c: No bootstrap required, opened a new log
I20260926 05:55:26.524307 13320 ts_tablet_manager.cc:1397] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:55:26.525606 13318 raft_consensus.cc:348] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7207a9da97c4522a959146d91a35095" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 35393 } } peers { permanent_uuid: "896c38a41da64789aa78026e5e19770f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39555 } } peers { permanent_uuid: "1d57f5a9d7084952badee6ab559f5b3c" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41987 } }
I20260926 05:55:26.525786 13318 raft_consensus.cc:374] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:26.525822 13318 raft_consensus.cc:729] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896c38a41da64789aa78026e5e19770f, State: Initialized, Role: FOLLOWER
I20260926 05:55:26.525966 13318 consensus_queue.cc:260] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7207a9da97c4522a959146d91a35095" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 35393 } } peers { permanent_uuid: "896c38a41da64789aa78026e5e19770f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39555 } } peers { permanent_uuid: "1d57f5a9d7084952badee6ab559f5b3c" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41987 } }
I20260926 05:55:26.526242 13318 ts_tablet_manager.cc:1428] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260926 05:55:26.526386 13042 heartbeater.cc:503] Master 127.24.255.62:38527 was elected leader, sending a full tablet report...
I20260926 05:55:26.526463 13320 raft_consensus.cc:348] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7207a9da97c4522a959146d91a35095" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 35393 } } peers { permanent_uuid: "896c38a41da64789aa78026e5e19770f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39555 } } peers { permanent_uuid: "1d57f5a9d7084952badee6ab559f5b3c" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41987 } }
I20260926 05:55:26.526598 13320 raft_consensus.cc:374] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:26.526630 13320 raft_consensus.cc:729] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d57f5a9d7084952badee6ab559f5b3c, State: Initialized, Role: FOLLOWER
I20260926 05:55:26.526578 13319 raft_consensus.cc:348] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7207a9da97c4522a959146d91a35095" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 35393 } } peers { permanent_uuid: "896c38a41da64789aa78026e5e19770f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39555 } } peers { permanent_uuid: "1d57f5a9d7084952badee6ab559f5b3c" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41987 } }
I20260926 05:55:26.526706 13319 raft_consensus.cc:374] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:26.526744 13319 raft_consensus.cc:729] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7207a9da97c4522a959146d91a35095, State: Initialized, Role: FOLLOWER
I20260926 05:55:26.526749 13320 consensus_queue.cc:260] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7207a9da97c4522a959146d91a35095" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 35393 } } peers { permanent_uuid: "896c38a41da64789aa78026e5e19770f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39555 } } peers { permanent_uuid: "1d57f5a9d7084952badee6ab559f5b3c" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41987 } }
I20260926 05:55:26.526857 13319 consensus_queue.cc:260] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7207a9da97c4522a959146d91a35095" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 35393 } } peers { permanent_uuid: "896c38a41da64789aa78026e5e19770f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39555 } } peers { permanent_uuid: "1d57f5a9d7084952badee6ab559f5b3c" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41987 } }
I20260926 05:55:26.527176 13320 ts_tablet_manager.cc:1428] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:55:26.527180 13319 ts_tablet_manager.cc:1428] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:55:26.527300 13173 heartbeater.cc:503] Master 127.24.255.62:38527 was elected leader, sending a full tablet report...
I20260926 05:55:26.527320 13304 heartbeater.cc:503] Master 127.24.255.62:38527 was elected leader, sending a full tablet report...
I20260926 05:55:26.569031 13325 raft_consensus.cc:482] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:55:26.569168 13325 raft_consensus.cc:504] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7207a9da97c4522a959146d91a35095" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 35393 } } peers { permanent_uuid: "896c38a41da64789aa78026e5e19770f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39555 } } peers { permanent_uuid: "1d57f5a9d7084952badee6ab559f5b3c" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41987 } }
I20260926 05:55:26.569574 13325 leader_election.cc:288] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7207a9da97c4522a959146d91a35095 (127.24.255.3:35393), 896c38a41da64789aa78026e5e19770f (127.24.255.1:39555)
I20260926 05:55:26.573125 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.569996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:60899 (local address 127.24.255.3:35393)
0926 05:55:26.570059 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:26.570061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:26.570178 (+ 117us) server_negotiation.cc:407] Connection header received
0926 05:55:26.570230 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:26.570231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:26.570241 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:26.570293 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:26.570469 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.570974 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.572252 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.572753 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.572825 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.572835 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:26.573012 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 05:55:26.573055 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:55:26.573200 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.569894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.3:35393 (local address 127.24.255.2:60899)
0926 05:55:26.570074 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:26.570083 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:26.570182 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:26.570304 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:26.570305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:26.570307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:26.570369 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.570371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.571051 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.571052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.572112 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.572119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.572871 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.572872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.572940 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.572943 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:55:26.572989 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:55:26.573046 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260926 05:55:26.573423 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.569854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.255.2:55491 (local address 127.24.255.1:39555)
0926 05:55:26.569915 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:26.569917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:26.569928 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:55:26.569972 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:26.569973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:26.569983 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:26.570030 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:55:26.570253 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.570840 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.572025 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.572616 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.572701 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.572714 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:26.573269 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 05:55:26.573353 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:55:26.573199 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.569785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:39555 (local address 127.24.255.2:55491)
0926 05:55:26.569849 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:26.569862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:26.569905 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:26.570075 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:26.570077 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:26.570079 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:55:26.570173 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.570176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.570934 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.570935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.571865 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:26.571868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.572842 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.572843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:26.572922 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.572926 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:55:26.572986 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:55:26.573040 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 05:55:26.573859 13258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4718eb1793436cae9a5d9b745c50ea" candidate_uuid: "1d57f5a9d7084952badee6ab559f5b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7207a9da97c4522a959146d91a35095" is_pre_election: true
I20260926 05:55:26.573935 12996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4718eb1793436cae9a5d9b745c50ea" candidate_uuid: "1d57f5a9d7084952badee6ab559f5b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896c38a41da64789aa78026e5e19770f" is_pre_election: true
I20260926 05:55:26.573993 13258 raft_consensus.cc:2454] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d57f5a9d7084952badee6ab559f5b3c in term 0.
I20260926 05:55:26.574159 12996 raft_consensus.cc:2454] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d57f5a9d7084952badee6ab559f5b3c in term 0.
I20260926 05:55:26.574261 13062 leader_election.cc:302] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d57f5a9d7084952badee6ab559f5b3c, b7207a9da97c4522a959146d91a35095; no voters:
I20260926 05:55:26.574398 13325 raft_consensus.cc:2781] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:55:26.574522 13325 raft_consensus.cc:482] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:55:26.574558 13325 raft_consensus.cc:3037] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:26.575285 13325 raft_consensus.cc:504] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7207a9da97c4522a959146d91a35095" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 35393 } } peers { permanent_uuid: "896c38a41da64789aa78026e5e19770f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39555 } } peers { permanent_uuid: "1d57f5a9d7084952badee6ab559f5b3c" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41987 } }
I20260926 05:55:26.575417 13325 leader_election.cc:288] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [CANDIDATE]: Term 1 election: Requested vote from peers b7207a9da97c4522a959146d91a35095 (127.24.255.3:35393), 896c38a41da64789aa78026e5e19770f (127.24.255.1:39555)
I20260926 05:55:26.575621 12996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4718eb1793436cae9a5d9b745c50ea" candidate_uuid: "1d57f5a9d7084952badee6ab559f5b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896c38a41da64789aa78026e5e19770f"
I20260926 05:55:26.575629 13258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4718eb1793436cae9a5d9b745c50ea" candidate_uuid: "1d57f5a9d7084952badee6ab559f5b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7207a9da97c4522a959146d91a35095"
I20260926 05:55:26.575695 12996 raft_consensus.cc:3037] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:26.575706 13258 raft_consensus.cc:3037] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:26.576416 12996 raft_consensus.cc:2454] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d57f5a9d7084952badee6ab559f5b3c in term 1.
I20260926 05:55:26.576527 13258 raft_consensus.cc:2454] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d57f5a9d7084952badee6ab559f5b3c in term 1.
I20260926 05:55:26.576620 13060 leader_election.cc:302] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d57f5a9d7084952badee6ab559f5b3c, 896c38a41da64789aa78026e5e19770f; no voters:
I20260926 05:55:26.576728 13325 raft_consensus.cc:2781] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:26.576882 13325 raft_consensus.cc:686] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [term 1 LEADER]: Becoming Leader. State: Replica: 1d57f5a9d7084952badee6ab559f5b3c, State: Running, Role: LEADER
I20260926 05:55:26.576978 13325 consensus_queue.cc:237] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7207a9da97c4522a959146d91a35095" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 35393 } } peers { permanent_uuid: "896c38a41da64789aa78026e5e19770f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39555 } } peers { permanent_uuid: "1d57f5a9d7084952badee6ab559f5b3c" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41987 } }
I20260926 05:55:26.578075 12854 catalog_manager.cc:5495] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d57f5a9d7084952badee6ab559f5b3c (127.24.255.2). New cstate: current_term: 1 leader_uuid: "1d57f5a9d7084952badee6ab559f5b3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7207a9da97c4522a959146d91a35095" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 35393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "896c38a41da64789aa78026e5e19770f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d57f5a9d7084952badee6ab559f5b3c" member_type: VOTER last_known_addr { host: "127.24.255.2" port: 41987 } health_report { overall_health: HEALTHY } } }
I20260926 05:55:26.592917 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.590363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.24.255.2:41987)
0926 05:55:26.590447 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:26.590450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:26.590461 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:55:26.590533 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:26.590534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:26.590558 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:26.590630 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:26.590890 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.591400 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.592176 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:26.592356 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:26.592410 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:26.592564 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:26.592672 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:26.592747 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:26.592786 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 05:55:26.592807 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:55:26.594866 13258 raft_consensus.cc:1261] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095 [term 1 FOLLOWER]: Refusing update from remote peer 1d57f5a9d7084952badee6ab559f5b3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:26.594866 12996 raft_consensus.cc:1261] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f [term 1 FOLLOWER]: Refusing update from remote peer 1d57f5a9d7084952badee6ab559f5b3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:26.595239 13328 consensus_queue.cc:1035] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "896c38a41da64789aa78026e5e19770f" member_type: VOTER last_known_addr { host: "127.24.255.1" port: 39555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:26.595397 13325 consensus_queue.cc:1035] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7207a9da97c4522a959146d91a35095" member_type: VOTER last_known_addr { host: "127.24.255.3" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:26.597746 13331 mvcc.cc:205] Tried to move back new op lower bound from 7333487110529896448 to 7333487110460047360. Current Snapshot: MvccSnapshot[applied={T|T < 7333487110529896448}]
I20260926 05:55:26.598269 13333 mvcc.cc:205] Tried to move back new op lower bound from 7333487110529896448 to 7333487110460047360. Current Snapshot: MvccSnapshot[applied={T|T < 7333487110529896448}]
I20260926 05:55:26.600603 12853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:55:26.600703 12853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:55:26.601506 12853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49fe0a1b812948c988b96f2408e0f870: Sending DeleteTablet for 3 replicas of tablet df4718eb1793436cae9a5d9b745c50ea
I20260926 05:55:26.601873 13238 tablet_service.cc:1514] Processing DeleteTablet for tablet df4718eb1793436cae9a5d9b745c50ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:26 UTC) from {username='slave'} at 127.0.0.1:55718
I20260926 05:55:26.601925 12976 tablet_service.cc:1514] Processing DeleteTablet for tablet df4718eb1793436cae9a5d9b745c50ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:26 UTC) from {username='slave'} at 127.0.0.1:56986
I20260926 05:55:26.601997 13107 tablet_service.cc:1514] Processing DeleteTablet for tablet df4718eb1793436cae9a5d9b745c50ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:55:26 UTC) from {username='slave'} at 127.0.0.1:44096
I20260926 05:55:26.602190 13342 tablet_replica.cc:330] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f: stopping tablet replica
I20260926 05:55:26.602293 13342 raft_consensus.cc:2229] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:55:26.602397 13342 raft_consensus.cc:2258] T df4718eb1793436cae9a5d9b745c50ea P 896c38a41da64789aa78026e5e19770f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:26.602614 13343 tablet_replica.cc:330] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c: stopping tablet replica
I20260926 05:55:26.602725 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 12914
I20260926 05:55:26.604646 13341 tablet_replica.cc:330] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095: stopping tablet replica
I20260926 05:55:26.604818 13341 raft_consensus.cc:2229] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:55:26.604939 13341 raft_consensus.cc:2258] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:26.605145 13343 raft_consensus.cc:2229] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:55:26.605298 13343 raft_consensus.cc:2258] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:55:26.605808 13343 ts_tablet_manager.cc:1905] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:55:26.607805 13343 ts_tablet_manager.cc:1918] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:55:26.607893 13343 log.cc:1196] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-1/wal/wals/df4718eb1793436cae9a5d9b745c50ea
I20260926 05:55:26.608206 13343 ts_tablet_manager.cc:1939] T df4718eb1793436cae9a5d9b745c50ea P 1d57f5a9d7084952badee6ab559f5b3c: Deleting consensus metadata
I20260926 05:55:26.608748 12839 catalog_manager.cc:4841] TS 1d57f5a9d7084952badee6ab559f5b3c (127.24.255.2:41987): tablet df4718eb1793436cae9a5d9b745c50ea (table test-table [id=3fde6c410acb477f8ca907c9bc238e17]) successfully deleted
I20260926 05:55:26.608830 13341 ts_tablet_manager.cc:1905] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 05:55:26.609071 12838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.255.1:39555 (error 108)
I20260926 05:55:26.609645 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 13045
I20260926 05:55:26.610647 13341 ts_tablet_manager.cc:1918] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:55:26.610718 13341 log.cc:1196] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095: Deleting WAL directory at /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401881171543-25596-0/minicluster-data/ts-2/wal/wals/df4718eb1793436cae9a5d9b745c50ea
I20260926 05:55:26.610934 13341 ts_tablet_manager.cc:1939] T df4718eb1793436cae9a5d9b745c50ea P b7207a9da97c4522a959146d91a35095: Deleting consensus metadata
I20260926 05:55:26.611056 13314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:26.610912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.1:39555 (local address 127.0.0.1:56994)
0926 05:55:26.610970 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:26.611017 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.1:39555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 05:55:26.611334 12838 catalog_manager.cc:4584] TS 896c38a41da64789aa78026e5e19770f (127.24.255.1:39555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df4718eb1793436cae9a5d9b745c50ea: Network error: Client connection negotiation failed: client connection to 127.24.255.1:39555: connect: Connection refused (error 111)
I20260926 05:55:26.613301 12840 catalog_manager.cc:4841] TS b7207a9da97c4522a959146d91a35095 (127.24.255.3:35393): tablet df4718eb1793436cae9a5d9b745c50ea (table test-table [id=3fde6c410acb477f8ca907c9bc238e17]) successfully deleted
I20260926 05:55:26.614771 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 13176
I20260926 05:55:26.619314 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 12823
2026-09-26T05:55:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (732 ms)
[----------] 6 tests from EncryptionPolicyTest (8071 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:26 dist-test-slave-7fr8 krb5kdc[13347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:26 dist-test-slave-7fr8 krb5kdc[13347](info): set up 2 sockets
Sep 26 05:55:26 dist-test-slave-7fr8 krb5kdc[13347](info): commencing operation
krb5kdc: starting...
W20260926 05:55:27.590502 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:27 dist-test-slave-7fr8 krb5kdc[13347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:55:27.722317 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:27.813297 13360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:27.813544 13360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:27.813570 13360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:27.814905 13360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:27.814965 13360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:27.814980 13360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:27.814990 13360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:27.815002 13360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:27.816874 13360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:27.817189 13360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:27.817458 13360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:27.822835 13368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:27.822770 13365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:27.822763 13366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:27.823212 13360 server_base.cc:1034] running on GCE node
I20260926 05:55:27.823496 13360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:27.823814 13360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:27.824987 13360 hybrid_clock.cc:648] HybridClock initialized: now 1790402127824946 us; error 64 us; skew 500 ppm
Sep 26 05:55:27 dist-test-slave-7fr8 krb5kdc[13347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:27.827847 13360 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:55:27.828372 13360 webserver.cc:460] Webserver started at http://127.0.0.1:41035/ using document root <none> and password file <none>
I20260926 05:55:27.828605 13360 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:27.828656 13360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:27.828789 13360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:27.829881 13360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "9935954a2f0d47cc995c62b0518a2ee1"
format_stamp: "Formatted at 2026-09-26 05:55:27 on dist-test-slave-7fr8"
server_key: "9a5ba5d2a8bb193a2b4e4a404b689486"
server_key_iv: "e9eebc59afac6df571e899d0f5d41b27"
server_key_version: "encryptionkey@0"
I20260926 05:55:27.830291 13360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "9935954a2f0d47cc995c62b0518a2ee1"
format_stamp: "Formatted at 2026-09-26 05:55:27 on dist-test-slave-7fr8"
server_key: "9a5ba5d2a8bb193a2b4e4a404b689486"
server_key_iv: "e9eebc59afac6df571e899d0f5d41b27"
server_key_version: "encryptionkey@0"
I20260926 05:55:27.831667 13360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:55:27.832736 13375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:27.833019 13360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:55:27.833141 13360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "9935954a2f0d47cc995c62b0518a2ee1"
format_stamp: "Formatted at 2026-09-26 05:55:27 on dist-test-slave-7fr8"
server_key: "9a5ba5d2a8bb193a2b4e4a404b689486"
server_key_iv: "e9eebc59afac6df571e899d0f5d41b27"
server_key_version: "encryptionkey@0"
I20260926 05:55:27.833214 13360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:27.843710 13360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:27.847107 13360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:27.847278 13360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:27.851572 13360 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42353
I20260926 05:55:27.851630 13427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42353 every 8 connection(s)
I20260926 05:55:27.851979 13360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:27.852864 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:27.856032 13428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1: Bootstrap starting.
I20260926 05:55:27.856613 13428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:27.856840 13428 log.cc:824] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:27.857600 13428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1: No bootstrap required, opened a new log
I20260926 05:55:27.857756 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 13360
I20260926 05:55:27.857861 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:27.858068 25596 external_mini_cluster.cc:1428] Setting key b0718ff8829133100164606a6142beac
I20260926 05:55:27.859853 13428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9935954a2f0d47cc995c62b0518a2ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260926 05:55:27.860050 13428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:27.860087 13428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9935954a2f0d47cc995c62b0518a2ee1, State: Initialized, Role: FOLLOWER
I20260926 05:55:27.860251 13428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9935954a2f0d47cc995c62b0518a2ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260926 05:55:27.860339 13428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:27.860369 13428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:27.860417 13428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:27.861629 13428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9935954a2f0d47cc995c62b0518a2ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260926 05:55:27.861752 13428 leader_election.cc:302] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9935954a2f0d47cc995c62b0518a2ee1; no voters:
I20260926 05:55:27.861917 13428 leader_election.cc:288] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:27.861991 13433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:27.862192 13428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:27.862248 13433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [term 1 LEADER]: Becoming Leader. State: Replica: 9935954a2f0d47cc995c62b0518a2ee1, State: Running, Role: LEADER
I20260926 05:55:27.862363 13433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9935954a2f0d47cc995c62b0518a2ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260926 05:55:27.862807 13435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9935954a2f0d47cc995c62b0518a2ee1. Latest consensus state: current_term: 1 leader_uuid: "9935954a2f0d47cc995c62b0518a2ee1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9935954a2f0d47cc995c62b0518a2ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } } }
I20260926 05:55:27.862884 13435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:27.863291 13434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9935954a2f0d47cc995c62b0518a2ee1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9935954a2f0d47cc995c62b0518a2ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } } }
I20260926 05:55:27.863359 13434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:27.863698 13442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:27.864301 13442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:27.865909 13442 catalog_manager.cc:1347] Generated new cluster ID: 096fcc91e8214b1a93dcef89ec9a53fa
I20260926 05:55:27.865964 13442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:55:27 dist-test-slave-7fr8 krb5kdc[13347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 05:55:27.871476 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:27.858713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58389 (local address 127.0.0.1:42353)
0926 05:55:27.858933 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:27.858946 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:27.858992 (+ 46us) server_negotiation.cc:407] Connection header received
0926 05:55:27.859809 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:27.859818 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:27.859891 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:27.860060 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:27.860860 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:27.862356 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:27.863524 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:27.864143 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:27.864253 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:27.867154 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:27.867159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:27.867160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:27.867174 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:27.869187 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:27.869657 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:27.869660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:27.869661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:27.869804 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:27.870015 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:27.870016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:27.870016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:27.870419 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:27.871030 (+ 611us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:27.871123 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 05:55:27.871246 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260926 05:55:27.872918 13442 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:27.872937 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:27.873418 13442 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:27.879737 13442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9935954a2f0d47cc995c62b0518a2ee1: Generated new TSK 0
I20260926 05:55:27.879956 13442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:27.887230 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:27.878639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60437 (local address 127.0.0.1:42353)
0926 05:55:27.878686 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:27.878689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:27.878962 (+ 273us) server_negotiation.cc:407] Connection header received
0926 05:55:27.878997 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:27.878999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:27.879011 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:27.879076 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:27.879774 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:27.880423 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:27.881817 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:27.882059 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:27.882191 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:27.883708 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:27.883709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:27.883709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:27.883717 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:27.885782 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:27.886338 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:27.886340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:27.886340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:27.886406 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:27.886711 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:27.886712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:27.886713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:27.886848 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:27.886946 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:27.887098 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 05:55:27.887157 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:55:27.889984 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 13360
W20260926 05:55:27.895303 13345 connection.cc:535] client connection to 127.0.0.1:42353 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42353): Connection reset by peer (error 104)
2026-09-26T05:55:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1270 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:27 dist-test-slave-7fr8 krb5kdc[13459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:27 dist-test-slave-7fr8 krb5kdc[13459](info): set up 2 sockets
Sep 26 05:55:27 dist-test-slave-7fr8 krb5kdc[13459](info): commencing operation
krb5kdc: starting...
W20260926 05:55:28.878947 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:28 dist-test-slave-7fr8 krb5kdc[13459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:55:29.027143 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:29.118849 13472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:29.119095 13472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:29.119122 13472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:29.120386 13472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:29.120445 13472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:29.120460 13472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:29.120471 13472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:29.120481 13472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:29.122164 13472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46429
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:29.122447 13472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:29.122686 13472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:29.128337 13480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:29.128373 13478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:29.128381 13477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:29.128504 13472 server_base.cc:1034] running on GCE node
I20260926 05:55:29.128912 13472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:29.129240 13472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:29.130416 13472 hybrid_clock.cc:648] HybridClock initialized: now 1790402129130365 us; error 63 us; skew 500 ppm
Sep 26 05:55:29 dist-test-slave-7fr8 krb5kdc[13459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:29.138304 13472 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:55:29.138837 13472 webserver.cc:460] Webserver started at http://127.0.0.1:37331/ using document root <none> and password file <none>
I20260926 05:55:29.139087 13472 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:29.139138 13472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:29.139262 13472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:29.140336 13472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "c1224da8a3a2405bab90345ac19943d0"
format_stamp: "Formatted at 2026-09-26 05:55:29 on dist-test-slave-7fr8"
server_key: "340ffe3b7ee3c81cbbb8b7cacc6d7ffa"
server_key_iv: "57d6dfe7c073f7b50f3679bf68e88252"
server_key_version: "encryptionkey@0"
I20260926 05:55:29.140668 13472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "c1224da8a3a2405bab90345ac19943d0"
format_stamp: "Formatted at 2026-09-26 05:55:29 on dist-test-slave-7fr8"
server_key: "340ffe3b7ee3c81cbbb8b7cacc6d7ffa"
server_key_iv: "57d6dfe7c073f7b50f3679bf68e88252"
server_key_version: "encryptionkey@0"
I20260926 05:55:29.142220 13472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:55:29.143559 13487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:29.143918 13472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:29.144095 13472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "c1224da8a3a2405bab90345ac19943d0"
format_stamp: "Formatted at 2026-09-26 05:55:29 on dist-test-slave-7fr8"
server_key: "340ffe3b7ee3c81cbbb8b7cacc6d7ffa"
server_key_iv: "57d6dfe7c073f7b50f3679bf68e88252"
server_key_version: "encryptionkey@0"
I20260926 05:55:29.144306 13472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:29.153231 13472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:29.156526 13472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:29.156764 13472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:29.161350 13472 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46429
I20260926 05:55:29.161441 13539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46429 every 8 connection(s)
I20260926 05:55:29.161775 13472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:29.162753 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:29.163381 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 13472
I20260926 05:55:29.163733 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:29.163960 25596 external_mini_cluster.cc:1428] Setting key 1e25d41154c9e23691929de0e64755d0
I20260926 05:55:29.165580 13540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0: Bootstrap starting.
I20260926 05:55:29.166190 13540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:29.166460 13540 log.cc:824] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:29.167232 13540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0: No bootstrap required, opened a new log
I20260926 05:55:29.170079 13540 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1224da8a3a2405bab90345ac19943d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46429 } }
I20260926 05:55:29.170380 13540 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:29.170413 13540 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1224da8a3a2405bab90345ac19943d0, State: Initialized, Role: FOLLOWER
I20260926 05:55:29.170578 13540 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1224da8a3a2405bab90345ac19943d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46429 } }
I20260926 05:55:29.170667 13540 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:29.170706 13540 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:29.170768 13540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:29.171283 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:29.164618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58429 (local address 127.0.0.1:46429)
0926 05:55:29.164849 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:29.164859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:29.164886 (+ 27us) server_negotiation.cc:407] Connection header received
0926 05:55:29.165405 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:29.165412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:29.165464 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:29.165609 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:29.165843 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:29.167000 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:29.168226 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:29.168911 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:29.169065 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:29.170236 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:29.170242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:29.170242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:29.170466 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:29.170535 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:29.170555 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:29.170667 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:29.170981 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 05:55:29.171025 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260926 05:55:29.172017 13540 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1224da8a3a2405bab90345ac19943d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46429 } }
I20260926 05:55:29.172173 13540 leader_election.cc:302] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1224da8a3a2405bab90345ac19943d0; no voters:
I20260926 05:55:29.172359 13540 leader_election.cc:288] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:29.172405 13545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:29.172556 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:29.172588 13545 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [term 1 LEADER]: Becoming Leader. State: Replica: c1224da8a3a2405bab90345ac19943d0, State: Running, Role: LEADER
I20260926 05:55:29.172631 13540 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:29.172676 13545 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1224da8a3a2405bab90345ac19943d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46429 } }
I20260926 05:55:29.173062 13546 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1224da8a3a2405bab90345ac19943d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1224da8a3a2405bab90345ac19943d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46429 } } }
I20260926 05:55:29.173019 13545 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1224da8a3a2405bab90345ac19943d0. Latest consensus state: current_term: 1 leader_uuid: "c1224da8a3a2405bab90345ac19943d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1224da8a3a2405bab90345ac19943d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46429 } } }
I20260926 05:55:29.173156 13546 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:29.173580 13545 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:29.173883 13554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:29.174464 13554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:29.177837 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:29.174701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55579 (local address 127.0.0.1:46429)
0926 05:55:29.174738 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:29.174740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:29.174881 (+ 141us) server_negotiation.cc:407] Connection header received
0926 05:55:29.174926 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:29.174927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:29.174938 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:29.175024 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:29.175170 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:29.175677 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:29.176393 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:29.176610 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:29.176666 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:29.177523 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:29.177525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:29.177525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:29.177579 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:29.177594 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:29.177599 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:29.177649 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:29.177741 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 05:55:29.177753 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 05:55:29.179443 13554 catalog_manager.cc:1347] Generated new cluster ID: 1971bfbeda10411984c8ad3769fc0f88
I20260926 05:55:29.179512 13554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:29.202271 13554 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:29.202942 13554 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:29.206080 13554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1224da8a3a2405bab90345ac19943d0: Generated new TSK 0
I20260926 05:55:29.206524 13554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:29.209728 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 13472
W20260926 05:55:29.213627 13457 connection.cc:535] client connection to 127.0.0.1:46429 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46429): Connection reset by peer (error 104)
2026-09-26T05:55:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:29 dist-test-slave-7fr8 krb5kdc[13571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:29 dist-test-slave-7fr8 krb5kdc[13571](info): set up 2 sockets
Sep 26 05:55:29 dist-test-slave-7fr8 krb5kdc[13571](info): commencing operation
krb5kdc: starting...
W20260926 05:55:30.190418 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:30 dist-test-slave-7fr8 krb5kdc[13571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:55:30.322057 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:30.406853 13584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:30.407090 13584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:30.407125 13584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:30.408566 13584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:30.408640 13584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:30.408661 13584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:30.408681 13584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:30.408695 13584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:30.410548 13584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:30.410836 13584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:30.411064 13584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:30.417193 13589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:30.417393 13590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:30.417527 13584 server_base.cc:1034] running on GCE node
W20260926 05:55:30.417471 13592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:30.418004 13584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:30.418363 13584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:30.419557 13584 hybrid_clock.cc:648] HybridClock initialized: now 1790402130419486 us; error 87 us; skew 500 ppm
Sep 26 05:55:30 dist-test-slave-7fr8 krb5kdc[13571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:30.422324 13584 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:55:30.422818 13584 webserver.cc:460] Webserver started at http://127.0.0.1:33577/ using document root <none> and password file <none>
I20260926 05:55:30.423055 13584 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:30.423110 13584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:30.423251 13584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:30.424405 13584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "b0842814772749bcab0d94f482b65d0a"
format_stamp: "Formatted at 2026-09-26 05:55:30 on dist-test-slave-7fr8"
server_key: "c65f5a8494e8b409497c0fe5c2b2adcd"
server_key_iv: "20b56bc474d89c774368e639a26aa23e"
server_key_version: "encryptionkey@0"
I20260926 05:55:30.424777 13584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "b0842814772749bcab0d94f482b65d0a"
format_stamp: "Formatted at 2026-09-26 05:55:30 on dist-test-slave-7fr8"
server_key: "c65f5a8494e8b409497c0fe5c2b2adcd"
server_key_iv: "20b56bc474d89c774368e639a26aa23e"
server_key_version: "encryptionkey@0"
I20260926 05:55:30.426349 13584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:55:30.427305 13599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:30.427661 13584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:30.427764 13584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "b0842814772749bcab0d94f482b65d0a"
format_stamp: "Formatted at 2026-09-26 05:55:30 on dist-test-slave-7fr8"
server_key: "c65f5a8494e8b409497c0fe5c2b2adcd"
server_key_iv: "20b56bc474d89c774368e639a26aa23e"
server_key_version: "encryptionkey@0"
I20260926 05:55:30.427835 13584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:30.431263 13584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:30.433406 13584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:30.433550 13584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:30.437896 13584 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40091
I20260926 05:55:30.437963 13651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40091 every 8 connection(s)
I20260926 05:55:30.438520 13584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:30.439774 13652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:30.442502 13652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a: Bootstrap starting.
I20260926 05:55:30.443629 13652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:30.444221 13652 log.cc:824] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:30.445109 13652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a: No bootstrap required, opened a new log
I20260926 05:55:30.446928 13652 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0842814772749bcab0d94f482b65d0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40091 } }
I20260926 05:55:30.447093 13652 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:30.447151 13652 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0842814772749bcab0d94f482b65d0a, State: Initialized, Role: FOLLOWER
I20260926 05:55:30.447302 13652 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0842814772749bcab0d94f482b65d0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40091 } }
I20260926 05:55:30.447341 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 13584
I20260926 05:55:30.447382 13652 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:30.447430 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:30.447417 13652 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:30.447527 13652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:30.447619 25596 external_mini_cluster.cc:1428] Setting key ec7570aebec29e23635625cfe89887e7
I20260926 05:55:30.448832 13652 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0842814772749bcab0d94f482b65d0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40091 } }
I20260926 05:55:30.448957 13652 leader_election.cc:302] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0842814772749bcab0d94f482b65d0a; no voters:
I20260926 05:55:30.449144 13652 leader_election.cc:288] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:30.449210 13657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:30.449366 13657 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [term 1 LEADER]: Becoming Leader. State: Replica: b0842814772749bcab0d94f482b65d0a, State: Running, Role: LEADER
I20260926 05:55:30.449390 13652 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:30.449524 13657 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0842814772749bcab0d94f482b65d0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40091 } }
I20260926 05:55:30.449922 13659 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0842814772749bcab0d94f482b65d0a. Latest consensus state: current_term: 1 leader_uuid: "b0842814772749bcab0d94f482b65d0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0842814772749bcab0d94f482b65d0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40091 } } }
I20260926 05:55:30.449934 13658 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0842814772749bcab0d94f482b65d0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0842814772749bcab0d94f482b65d0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40091 } } }
I20260926 05:55:30.450088 13659 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:30.450201 13658 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:30.450932 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:30.448165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54119 (local address 127.0.0.1:40091)
0926 05:55:30.448415 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:30.448427 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:30.448467 (+ 40us) server_negotiation.cc:407] Connection header received
0926 05:55:30.448595 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:30.448603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:30.448634 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:30.448788 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:30.450306 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:30.450310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:30.450310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:30.450315 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:30.450365 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:30.450382 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:30.450442 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:30.450682 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 05:55:30.450737 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":131,"threads_started":1}
W20260926 05:55:30.451321 13673 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:55:30.451364 13673 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:55:30.451414 13666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:30.451582 13666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:30.452564 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:30.454001 13666 catalog_manager.cc:1347] Generated new cluster ID: aa909732a0aa41109e2585235eb167f1
I20260926 05:55:30.454074 13666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:30.459955 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:30.458619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60191 (local address 127.0.0.1:40091)
0926 05:55:30.458671 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:30.458674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:30.458794 (+ 120us) server_negotiation.cc:407] Connection header received
0926 05:55:30.458875 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:30.458876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:30.458884 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:30.458940 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:30.459718 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:30.459720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:30.459720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:30.459725 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:30.459743 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:30.459748 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:30.459812 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:30.459861 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 05:55:30.459875 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 05:55:30.462967 13666 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:30.463158 13666 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:30.478876 13666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0842814772749bcab0d94f482b65d0a: Generated new TSK 0
I20260926 05:55:30.479106 13666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:30.485512 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 13584
2026-09-26T05:55:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1275 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:30 dist-test-slave-7fr8 krb5kdc[13683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:30 dist-test-slave-7fr8 krb5kdc[13683](info): set up 2 sockets
Sep 26 05:55:30 dist-test-slave-7fr8 krb5kdc[13683](info): commencing operation
krb5kdc: starting...
W20260926 05:55:31.462334 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:31 dist-test-slave-7fr8 krb5kdc[13683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:55:31.584594 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:31.667965 13696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:31.668221 13696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:31.668247 13696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:31.669448 13696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:31.669489 13696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:31.669502 13696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:31.669513 13696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:31.669524 13696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:31.671066 13696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:31.671288 13696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:31.671486 13696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:31.676518 13702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:31.676611 13701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:31.676608 13704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:31.676936 13696 server_base.cc:1034] running on GCE node
I20260926 05:55:31.677191 13696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:31.677474 13696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:31.678661 13696 hybrid_clock.cc:648] HybridClock initialized: now 1790402131678644 us; error 31 us; skew 500 ppm
Sep 26 05:55:31 dist-test-slave-7fr8 krb5kdc[13683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:31.681269 13696 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:55:31.681748 13696 webserver.cc:460] Webserver started at http://127.0.0.1:34675/ using document root <none> and password file <none>
I20260926 05:55:31.681953 13696 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:31.682000 13696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:31.682154 13696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:31.683306 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "1925425bfa2e465bb393fd5c8a8156a2"
format_stamp: "Formatted at 2026-09-26 05:55:31 on dist-test-slave-7fr8"
server_key: "04c90a69b561103a9111dd7dbddcefee"
server_key_iv: "8d0d76173a41247ad7ce52db3b89473a"
server_key_version: "encryptionkey@0"
I20260926 05:55:31.683676 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "1925425bfa2e465bb393fd5c8a8156a2"
format_stamp: "Formatted at 2026-09-26 05:55:31 on dist-test-slave-7fr8"
server_key: "04c90a69b561103a9111dd7dbddcefee"
server_key_iv: "8d0d76173a41247ad7ce52db3b89473a"
server_key_version: "encryptionkey@0"
I20260926 05:55:31.685000 13696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:55:31.685791 13711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:31.686023 13696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:55:31.686139 13696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "1925425bfa2e465bb393fd5c8a8156a2"
format_stamp: "Formatted at 2026-09-26 05:55:31 on dist-test-slave-7fr8"
server_key: "04c90a69b561103a9111dd7dbddcefee"
server_key_iv: "8d0d76173a41247ad7ce52db3b89473a"
server_key_version: "encryptionkey@0"
I20260926 05:55:31.686219 13696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:31.693070 13696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:31.695756 13696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:31.695909 13696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:31.700433 13696 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37879
I20260926 05:55:31.700589 13763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37879 every 8 connection(s)
I20260926 05:55:31.700845 13696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:31.702152 13764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:31.704794 13764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2: Bootstrap starting.
I20260926 05:55:31.705374 13764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:31.705613 13764 log.cc:824] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:31.706854 13764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2: No bootstrap required, opened a new log
I20260926 05:55:31.708822 13764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1925425bfa2e465bb393fd5c8a8156a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260926 05:55:31.709097 13764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:31.709125 13764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1925425bfa2e465bb393fd5c8a8156a2, State: Initialized, Role: FOLLOWER
I20260926 05:55:31.709384 13764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1925425bfa2e465bb393fd5c8a8156a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260926 05:55:31.709506 13764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:31.709537 13764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:31.709573 13764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:31.710002 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 13696
I20260926 05:55:31.710145 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:31.710544 25596 external_mini_cluster.cc:1428] Setting key 2ee320439f4b3a10bb3bf75797f6c5c4
I20260926 05:55:31.710891 13764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1925425bfa2e465bb393fd5c8a8156a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260926 05:55:31.711057 13764 leader_election.cc:302] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1925425bfa2e465bb393fd5c8a8156a2; no voters:
I20260926 05:55:31.711253 13764 leader_election.cc:288] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:31.711344 13768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:31.711589 13768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [term 1 LEADER]: Becoming Leader. State: Replica: 1925425bfa2e465bb393fd5c8a8156a2, State: Running, Role: LEADER
I20260926 05:55:31.711728 13764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:31.711803 13768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1925425bfa2e465bb393fd5c8a8156a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } }
I20260926 05:55:31.712649 13770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1925425bfa2e465bb393fd5c8a8156a2. Latest consensus state: current_term: 1 leader_uuid: "1925425bfa2e465bb393fd5c8a8156a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1925425bfa2e465bb393fd5c8a8156a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } } }
I20260926 05:55:31.712752 13770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:31.713229 13771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1925425bfa2e465bb393fd5c8a8156a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1925425bfa2e465bb393fd5c8a8156a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37879 } } }
I20260926 05:55:31.713399 13771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2 [sys.catalog]: This master's current role is: LEADER
W20260926 05:55:31.713900 13785 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:55:31.713946 13785 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:55:31.714039 13778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:31.714223 13778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:31.715955 13778 catalog_manager.cc:1347] Generated new cluster ID: 118826f82dd444ff8eda4521eb874f09
I20260926 05:55:31.716053 13778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:55:31 dist-test-slave-7fr8 krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 05:55:31.722718 13778 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:31.723487 13778 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:31.723680 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:31.711144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43273 (local address 127.0.0.1:37879)
0926 05:55:31.711347 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:31.711360 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:31.711409 (+ 49us) server_negotiation.cc:407] Connection header received
0926 05:55:31.711539 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:31.711545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:31.711601 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:31.711775 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:31.712598 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:31.714337 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:31.715228 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:31.715832 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:31.715940 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:31.718856 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:31.718862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:31.718862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:31.718878 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:31.721588 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:31.722043 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:31.722047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:31.722047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:31.722195 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:31.722452 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:31.722453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:31.722453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:31.722765 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:31.723279 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:31.723342 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 05:55:31.723457 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260926 05:55:31.724546 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:31.728001 13778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1925425bfa2e465bb393fd5c8a8156a2: Generated new TSK 0
I20260926 05:55:31.728230 13778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:31.732864 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:31.725866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54937 (local address 127.0.0.1:37879)
0926 05:55:31.725906 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:31.725908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:31.725971 (+ 63us) server_negotiation.cc:407] Connection header received
0926 05:55:31.726031 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:31.726032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:31.726043 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:31.726087 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:31.726630 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:31.727287 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:31.728531 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:31.728780 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:31.728842 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:31.730078 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:31.730079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:31.730080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:31.730087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:31.731805 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:31.732167 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:31.732168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:31.732169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:31.732228 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:31.732382 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:31.732382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:31.732383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:31.732497 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:31.732587 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:31.732690 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 05:55:31.732771 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:55:31.734884 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 13696
2026-09-26T05:55:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:31 dist-test-slave-7fr8 krb5kdc[13795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:31 dist-test-slave-7fr8 krb5kdc[13795](info): set up 2 sockets
Sep 26 05:55:31 dist-test-slave-7fr8 krb5kdc[13795](info): commencing operation
krb5kdc: starting...
W20260926 05:55:32.722191 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:32 dist-test-slave-7fr8 krb5kdc[13795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:55:32.849602 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:32.931064 13808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:32.931303 13808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:32.931334 13808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:32.932596 13808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:32.932649 13808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:32.932667 13808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:32.932684 13808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:32.932698 13808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:32.934357 13808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:32.934643 13808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:32.934856 13808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:32.940280 13816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:32.940214 13814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:32.940230 13808 server_base.cc:1034] running on GCE node
W20260926 05:55:32.940222 13813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:32.940718 13808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:32.941067 13808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:32.942200 13808 hybrid_clock.cc:648] HybridClock initialized: now 1790402132942138 us; error 84 us; skew 500 ppm
Sep 26 05:55:32 dist-test-slave-7fr8 krb5kdc[13795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:32.945071 13808 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:55:32.945574 13808 webserver.cc:460] Webserver started at http://127.0.0.1:37477/ using document root <none> and password file <none>
I20260926 05:55:32.945799 13808 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:32.945848 13808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:32.945979 13808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:32.947799 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "ea7f4556e6a849a59635a7555cc2b503"
format_stamp: "Formatted at 2026-09-26 05:55:32 on dist-test-slave-7fr8"
server_key: "907b3f5cbbef5e57bd4f8fa382031b91"
server_key_iv: "db7264f721c9ffbeed32d89649195ba7"
server_key_version: "encryptionkey@0"
I20260926 05:55:32.948275 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "ea7f4556e6a849a59635a7555cc2b503"
format_stamp: "Formatted at 2026-09-26 05:55:32 on dist-test-slave-7fr8"
server_key: "907b3f5cbbef5e57bd4f8fa382031b91"
server_key_iv: "db7264f721c9ffbeed32d89649195ba7"
server_key_version: "encryptionkey@0"
I20260926 05:55:32.949712 13808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:55:32.950604 13823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:32.950788 13808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:55:32.950878 13808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "ea7f4556e6a849a59635a7555cc2b503"
format_stamp: "Formatted at 2026-09-26 05:55:32 on dist-test-slave-7fr8"
server_key: "907b3f5cbbef5e57bd4f8fa382031b91"
server_key_iv: "db7264f721c9ffbeed32d89649195ba7"
server_key_version: "encryptionkey@0"
I20260926 05:55:32.950965 13808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:32.960719 13808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:32.963564 13808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:32.963732 13808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:32.968077 13808 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44315
I20260926 05:55:32.968122 13875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44315 every 8 connection(s)
I20260926 05:55:32.968474 13808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:32.969331 13876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:32.971722 13876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503: Bootstrap starting.
I20260926 05:55:32.972304 13876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:32.972560 13876 log.cc:824] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:32.973378 13876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503: No bootstrap required, opened a new log
I20260926 05:55:32.974916 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 13808
I20260926 05:55:32.975039 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:32.975267 25596 external_mini_cluster.cc:1428] Setting key ba51157691c5747d9765a589a82931bb
I20260926 05:55:32.975497 13876 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea7f4556e6a849a59635a7555cc2b503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260926 05:55:32.975685 13876 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:32.975759 13876 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea7f4556e6a849a59635a7555cc2b503, State: Initialized, Role: FOLLOWER
I20260926 05:55:32.975917 13876 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea7f4556e6a849a59635a7555cc2b503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260926 05:55:32.975999 13876 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:32.976055 13876 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:32.976141 13876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:32.977483 13876 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea7f4556e6a849a59635a7555cc2b503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260926 05:55:32.977754 13876 leader_election.cc:302] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea7f4556e6a849a59635a7555cc2b503; no voters:
I20260926 05:55:32.978003 13876 leader_election.cc:288] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:32.978125 13881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:32.978317 13876 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:32.978327 13881 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [term 1 LEADER]: Becoming Leader. State: Replica: ea7f4556e6a849a59635a7555cc2b503, State: Running, Role: LEADER
I20260926 05:55:32.978575 13881 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea7f4556e6a849a59635a7555cc2b503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260926 05:55:32.978983 13883 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea7f4556e6a849a59635a7555cc2b503. Latest consensus state: current_term: 1 leader_uuid: "ea7f4556e6a849a59635a7555cc2b503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea7f4556e6a849a59635a7555cc2b503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } } }
I20260926 05:55:32.979405 13883 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:32.979326 13882 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea7f4556e6a849a59635a7555cc2b503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea7f4556e6a849a59635a7555cc2b503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } } }
I20260926 05:55:32.979664 13882 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:32.979818 13895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:32.980415 13895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:32.981953 13895 catalog_manager.cc:1347] Generated new cluster ID: c0d283a27605402298daa2d805f67ec9
I20260926 05:55:32.982012 13895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:32.983228 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:32.975867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55161 (local address 127.0.0.1:44315)
0926 05:55:32.976138 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:32.976159 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:32.976274 (+ 115us) server_negotiation.cc:407] Connection header received
0926 05:55:32.976446 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:32.976454 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:32.976514 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:32.976680 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:32.976894 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:32.978484 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:32.979654 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:32.980289 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:32.980400 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:32.982561 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:32.982565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:32.982565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:32.982756 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:32.982794 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:32.982806 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:32.982863 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:32.982938 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 05:55:32.983010 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
I20260926 05:55:32.984124 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:32.988269 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:32.985535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35673 (local address 127.0.0.1:44315)
0926 05:55:32.985577 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:32.985579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:32.985684 (+ 105us) server_negotiation.cc:407] Connection header received
0926 05:55:32.985738 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:32.985739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:32.985751 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:32.985796 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:32.985925 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:32.986473 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:32.987159 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:32.987356 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:32.987411 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:32.988017 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:32.988018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:32.988018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:32.988023 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:32.988039 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:32.988043 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:32.988095 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:32.988154 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 05:55:32.988183 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:55:32.997927 13895 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:32.998726 13895 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:33.006814 13895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea7f4556e6a849a59635a7555cc2b503: Generated new TSK 0
I20260926 05:55:33.007138 13895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:33.009832 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 13808
2026-09-26T05:55:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1274 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:33 dist-test-slave-7fr8 krb5kdc[13907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:33 dist-test-slave-7fr8 krb5kdc[13907](info): set up 2 sockets
Sep 26 05:55:33 dist-test-slave-7fr8 krb5kdc[13907](info): commencing operation
krb5kdc: starting...
W20260926 05:55:33.982246 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:34 dist-test-slave-7fr8 krb5kdc[13907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:55:34.114951 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:34.206944 13920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:34.207309 13920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:34.207340 13920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:34.208676 13920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:34.208746 13920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:34.208761 13920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:34.208773 13920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:34.208784 13920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:34.210533 13920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:34.210807 13920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:34.211035 13920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:55:34.217131 13920 server_base.cc:1034] running on GCE node
W20260926 05:55:34.217129 13925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:34.217072 13926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:34.217103 13928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:34.217522 13920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:34.217818 13920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:34.219074 13920 hybrid_clock.cc:648] HybridClock initialized: now 1790402134219018 us; error 51 us; skew 500 ppm
Sep 26 05:55:34 dist-test-slave-7fr8 krb5kdc[13907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:34.228873 13920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:55:34.229360 13920 webserver.cc:460] Webserver started at http://127.0.0.1:39063/ using document root <none> and password file <none>
I20260926 05:55:34.229585 13920 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:34.229631 13920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:34.229724 13920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:34.230830 13920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "82fa766188fc4336b2cccd89bba8eec8"
format_stamp: "Formatted at 2026-09-26 05:55:34 on dist-test-slave-7fr8"
server_key: "1228b67ca78b7f05fc39a6c707361ed5"
server_key_iv: "834a6606cda52b226f1672bf2ef40b6a"
server_key_version: "encryptionkey@0"
I20260926 05:55:34.231164 13920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "82fa766188fc4336b2cccd89bba8eec8"
format_stamp: "Formatted at 2026-09-26 05:55:34 on dist-test-slave-7fr8"
server_key: "1228b67ca78b7f05fc39a6c707361ed5"
server_key_iv: "834a6606cda52b226f1672bf2ef40b6a"
server_key_version: "encryptionkey@0"
I20260926 05:55:34.232537 13920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:55:34.233467 13935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:34.233829 13920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:34.233953 13920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "82fa766188fc4336b2cccd89bba8eec8"
format_stamp: "Formatted at 2026-09-26 05:55:34 on dist-test-slave-7fr8"
server_key: "1228b67ca78b7f05fc39a6c707361ed5"
server_key_iv: "834a6606cda52b226f1672bf2ef40b6a"
server_key_version: "encryptionkey@0"
I20260926 05:55:34.234014 13920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:34.237488 13920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:34.239986 13920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:34.240120 13920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:34.244123 13920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34253
I20260926 05:55:34.244333 13987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34253 every 8 connection(s)
I20260926 05:55:34.244521 13920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:34.245615 13988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:34.248234 13988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8: Bootstrap starting.
I20260926 05:55:34.248822 13988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:34.249085 13988 log.cc:824] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:34.249866 13988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8: No bootstrap required, opened a new log
I20260926 05:55:34.251583 13988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fa766188fc4336b2cccd89bba8eec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260926 05:55:34.251739 13988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:34.251711 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 13920
I20260926 05:55:34.251778 13988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82fa766188fc4336b2cccd89bba8eec8, State: Initialized, Role: FOLLOWER
I20260926 05:55:34.251868 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:34.251927 13988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fa766188fc4336b2cccd89bba8eec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260926 05:55:34.252015 13988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:34.252043 13988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:34.252099 13988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:34.252126 25596 external_mini_cluster.cc:1428] Setting key 38029c568da1552fd6138ced2d1c34ff
I20260926 05:55:34.252786 13988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fa766188fc4336b2cccd89bba8eec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260926 05:55:34.252944 13988 leader_election.cc:302] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82fa766188fc4336b2cccd89bba8eec8; no voters:
I20260926 05:55:34.253140 13988 leader_election.cc:288] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:34.253185 13993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:34.253378 13993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [term 1 LEADER]: Becoming Leader. State: Replica: 82fa766188fc4336b2cccd89bba8eec8, State: Running, Role: LEADER
I20260926 05:55:34.253387 13988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:34.253547 13993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fa766188fc4336b2cccd89bba8eec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260926 05:55:34.253901 13994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82fa766188fc4336b2cccd89bba8eec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fa766188fc4336b2cccd89bba8eec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } } }
I20260926 05:55:34.253976 13994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:34.254184 13995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82fa766188fc4336b2cccd89bba8eec8. Latest consensus state: current_term: 1 leader_uuid: "82fa766188fc4336b2cccd89bba8eec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fa766188fc4336b2cccd89bba8eec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } } }
I20260926 05:55:34.254305 13995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:34.254765 13997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:34.255100 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:34.252840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40767 (local address 127.0.0.1:34253)
0926 05:55:34.253060 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:34.253071 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:34.253115 (+ 44us) server_negotiation.cc:407] Connection header received
0926 05:55:34.253253 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:34.253261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:34.253294 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:34.254199 (+ 905us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:34.254536 (+ 337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:34.254539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:34.254540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:34.254544 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:34.254586 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:34.254600 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:34.254667 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:34.254779 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 05:55:34.254825 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":102,"threads_started":1}
I20260926 05:55:34.255395 13997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:34.256032 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:34.257091 13997 catalog_manager.cc:1347] Generated new cluster ID: bcce91ba47ff40cda8f48fa7e307147b
I20260926 05:55:34.257134 13997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:34.267359 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:34.266186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41291 (local address 127.0.0.1:34253)
0926 05:55:34.266253 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:34.266255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:34.266271 (+ 16us) server_negotiation.cc:407] Connection header received
0926 05:55:34.266304 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:34.266306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:34.266314 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:34.266411 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:34.267120 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:34.267122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:34.267122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:34.267127 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:34.267150 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:34.267155 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:34.267218 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:34.267262 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 05:55:34.267276 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:55:34.270643 13997 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:34.270831 13997 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:34.280275 13997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82fa766188fc4336b2cccd89bba8eec8: Generated new TSK 0
I20260926 05:55:34.280501 13997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:34.284991 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 13920
2026-09-26T05:55:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1275 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:34 dist-test-slave-7fr8 krb5kdc[14019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:34 dist-test-slave-7fr8 krb5kdc[14019](info): set up 2 sockets
Sep 26 05:55:34 dist-test-slave-7fr8 krb5kdc[14019](info): commencing operation
krb5kdc: starting...
W20260926 05:55:35.250371 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:35 dist-test-slave-7fr8 krb5kdc[14019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:55:35.375416 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:35.458647 14032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:35.459003 14032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:35.459066 14032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:35.460510 14032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:35.460570 14032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:35.460585 14032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:35.460597 14032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:35.460608 14032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:35.462253 14032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:35.462478 14032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:35.462663 14032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:35.467502 14038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:35.467521 14037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:35.467502 14040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:35.467660 14032 server_base.cc:1034] running on GCE node
I20260926 05:55:35.468032 14032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:35.468343 14032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:35.469506 14032 hybrid_clock.cc:648] HybridClock initialized: now 1790402135469484 us; error 34 us; skew 500 ppm
Sep 26 05:55:35 dist-test-slave-7fr8 krb5kdc[14019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:35.471961 14032 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:55:35.472376 14032 webserver.cc:460] Webserver started at http://127.0.0.1:43445/ using document root <none> and password file <none>
I20260926 05:55:35.472586 14032 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:35.472632 14032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:35.472723 14032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:35.473766 14032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "ae6670732e974e6fa9e493410028fd35"
format_stamp: "Formatted at 2026-09-26 05:55:35 on dist-test-slave-7fr8"
server_key: "be2040d48901bfa67840a59f48cc3c3f"
server_key_iv: "67dd582027676029fbb85883ef5e515a"
server_key_version: "encryptionkey@0"
I20260926 05:55:35.474128 14032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "ae6670732e974e6fa9e493410028fd35"
format_stamp: "Formatted at 2026-09-26 05:55:35 on dist-test-slave-7fr8"
server_key: "be2040d48901bfa67840a59f48cc3c3f"
server_key_iv: "67dd582027676029fbb85883ef5e515a"
server_key_version: "encryptionkey@0"
I20260926 05:55:35.475576 14032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:35.476558 14047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:35.476756 14032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:35.476866 14032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "ae6670732e974e6fa9e493410028fd35"
format_stamp: "Formatted at 2026-09-26 05:55:35 on dist-test-slave-7fr8"
server_key: "be2040d48901bfa67840a59f48cc3c3f"
server_key_iv: "67dd582027676029fbb85883ef5e515a"
server_key_version: "encryptionkey@0"
I20260926 05:55:35.476938 14032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:35.496474 14032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:35.498888 14032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:35.499042 14032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:35.503054 14032 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43003
I20260926 05:55:35.503110 14099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43003 every 8 connection(s)
I20260926 05:55:35.503463 14032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:35.504352 14100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:35.506850 14100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35: Bootstrap starting.
I20260926 05:55:35.507472 14100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:35.507753 14100 log.cc:824] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:35.508921 14100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35: No bootstrap required, opened a new log
I20260926 05:55:35.510689 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 14032
I20260926 05:55:35.510798 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:35.511013 25596 external_mini_cluster.cc:1428] Setting key 940a6afea32b958c526a8fb562e61615
I20260926 05:55:35.511142 14100 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6670732e974e6fa9e493410028fd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } }
I20260926 05:55:35.511328 14100 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:35.511365 14100 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6670732e974e6fa9e493410028fd35, State: Initialized, Role: FOLLOWER
I20260926 05:55:35.511533 14100 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6670732e974e6fa9e493410028fd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } }
I20260926 05:55:35.511637 14100 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:35.511672 14100 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:35.511767 14100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:35.512919 14100 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6670732e974e6fa9e493410028fd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } }
I20260926 05:55:35.513038 14100 leader_election.cc:302] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae6670732e974e6fa9e493410028fd35; no voters:
I20260926 05:55:35.513197 14100 leader_election.cc:288] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:35.513253 14105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:35.513459 14105 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [term 1 LEADER]: Becoming Leader. State: Replica: ae6670732e974e6fa9e493410028fd35, State: Running, Role: LEADER
I20260926 05:55:35.513463 14100 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:35.513657 14105 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6670732e974e6fa9e493410028fd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } }
I20260926 05:55:35.514140 14106 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae6670732e974e6fa9e493410028fd35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6670732e974e6fa9e493410028fd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } } }
I20260926 05:55:35.514228 14106 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:35.514523 14107 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6670732e974e6fa9e493410028fd35. Latest consensus state: current_term: 1 leader_uuid: "ae6670732e974e6fa9e493410028fd35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6670732e974e6fa9e493410028fd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43003 } } }
I20260926 05:55:35.514642 14107 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:35.514813 14114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:35.515352 14114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 05:55:35 dist-test-slave-7fr8 krb5kdc[14019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 05:55:35.517786 14114 catalog_manager.cc:1347] Generated new cluster ID: 6bc7569385bd492594705d26a883ab2f
I20260926 05:55:35.517823 14114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:35.522312 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:35.511535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:44299 (local address 127.0.0.1:43003)
0926 05:55:35.511757 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:35.511767 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:35.511802 (+ 35us) server_negotiation.cc:407] Connection header received
0926 05:55:35.511939 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:35.511947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:35.512008 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:35.512175 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:35.512845 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:35.514079 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:35.515168 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:35.515745 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:35.515854 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:35.518342 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:35.518347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:35.518347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:35.518360 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:35.520338 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:35.520791 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:35.520795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:35.520795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:35.520924 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:35.521138 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:35.521139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:35.521140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:35.521511 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:35.521680 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:35.521934 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 05:55:35.522058 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260926 05:55:35.523270 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:35.530777 14114 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:35.531283 14114 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:35.532029 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:35.524850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:52635 (local address 127.0.0.1:43003)
0926 05:55:35.524883 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:35.524885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:35.524893 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:55:35.524916 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:35.524916 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:35.524926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:35.525002 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:35.525583 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:35.526420 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:35.527378 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:35.527717 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:35.527785 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:35.529151 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:35.529153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:35.529153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:35.529160 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:35.530704 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:35.531153 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:35.531155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:35.531155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:35.531211 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:35.531428 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:35.531429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:35.531429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:35.531550 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:35.531637 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:35.531868 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 05:55:35.531938 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 05:55:35.534504 14114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae6670732e974e6fa9e493410028fd35: Generated new TSK 0
I20260926 05:55:35.534672 14114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:35.542261 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 14032
W20260926 05:55:35.546284 14017 connection.cc:535] client connection to 127.0.0.1:43003 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43003): Connection reset by peer (error 104)
2026-09-26T05:55:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1257 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:35 dist-test-slave-7fr8 krb5kdc[14131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:35 dist-test-slave-7fr8 krb5kdc[14131](info): set up 2 sockets
Sep 26 05:55:35 dist-test-slave-7fr8 krb5kdc[14131](info): commencing operation
krb5kdc: starting...
W20260926 05:55:36.510897 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:36 dist-test-slave-7fr8 krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:55:36.646354 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:36.732375 14144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:36.732633 14144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:36.732671 14144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:36.733958 14144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:36.734016 14144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:36.734035 14144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:36.734052 14144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:36.734068 14144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:36.735736 14144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:36.736022 14144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:36.736255 14144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:36.741451 14149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:36.741429 14152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:36.741432 14150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:36.741492 14144 server_base.cc:1034] running on GCE node
I20260926 05:55:36.741948 14144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:36.742304 14144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:36.743502 14144 hybrid_clock.cc:648] HybridClock initialized: now 1790402136743464 us; error 51 us; skew 500 ppm
Sep 26 05:55:36 dist-test-slave-7fr8 krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:36.746179 14144 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:55:36.746645 14144 webserver.cc:460] Webserver started at http://127.0.0.1:45469/ using document root <none> and password file <none>
I20260926 05:55:36.746857 14144 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:36.746906 14144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:36.747031 14144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:36.748164 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "6d8c4e1a5fef409f9ad62883f2b4f860"
format_stamp: "Formatted at 2026-09-26 05:55:36 on dist-test-slave-7fr8"
server_key: "3a3d07a91e382cf3339ecc77047468f8"
server_key_iv: "dbc8ad955f4df952d9e710be76b3d0f7"
server_key_version: "encryptionkey@0"
I20260926 05:55:36.748505 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "6d8c4e1a5fef409f9ad62883f2b4f860"
format_stamp: "Formatted at 2026-09-26 05:55:36 on dist-test-slave-7fr8"
server_key: "3a3d07a91e382cf3339ecc77047468f8"
server_key_iv: "dbc8ad955f4df952d9e710be76b3d0f7"
server_key_version: "encryptionkey@0"
I20260926 05:55:36.749802 14144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:55:36.750720 14159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:36.750954 14144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:55:36.751053 14144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "6d8c4e1a5fef409f9ad62883f2b4f860"
format_stamp: "Formatted at 2026-09-26 05:55:36 on dist-test-slave-7fr8"
server_key: "3a3d07a91e382cf3339ecc77047468f8"
server_key_iv: "dbc8ad955f4df952d9e710be76b3d0f7"
server_key_version: "encryptionkey@0"
I20260926 05:55:36.751127 14144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:36.784955 14144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:36.787405 14144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:36.787616 14144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:36.791842 14144 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45867
I20260926 05:55:36.791930 14211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45867 every 8 connection(s)
I20260926 05:55:36.792603 14144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:36.793731 14212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:36.796531 14212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860: Bootstrap starting.
I20260926 05:55:36.797210 14212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:36.797504 14212 log.cc:824] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:36.798482 14212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860: No bootstrap required, opened a new log
I20260926 05:55:36.800264 14212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8c4e1a5fef409f9ad62883f2b4f860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45867 } }
I20260926 05:55:36.800426 14212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:36.800450 14212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8c4e1a5fef409f9ad62883f2b4f860, State: Initialized, Role: FOLLOWER
I20260926 05:55:36.800609 14212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8c4e1a5fef409f9ad62883f2b4f860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45867 } }
I20260926 05:55:36.800683 14212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:36.800704 14212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:36.800737 14212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:36.801367 14212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8c4e1a5fef409f9ad62883f2b4f860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45867 } }
I20260926 05:55:36.801462 14212 leader_election.cc:302] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d8c4e1a5fef409f9ad62883f2b4f860; no voters:
I20260926 05:55:36.801657 14212 leader_election.cc:288] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:36.801740 14215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:36.801898 14215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [term 1 LEADER]: Becoming Leader. State: Replica: 6d8c4e1a5fef409f9ad62883f2b4f860, State: Running, Role: LEADER
I20260926 05:55:36.801903 14212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:36.802119 14215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8c4e1a5fef409f9ad62883f2b4f860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45867 } }
I20260926 05:55:36.802377 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 14144
I20260926 05:55:36.802531 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:36.802724 25596 external_mini_cluster.cc:1428] Setting key 10172d83341206d919b4e65d2e5e42d2
I20260926 05:55:36.802772 14217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d8c4e1a5fef409f9ad62883f2b4f860. Latest consensus state: current_term: 1 leader_uuid: "6d8c4e1a5fef409f9ad62883f2b4f860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8c4e1a5fef409f9ad62883f2b4f860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45867 } } }
I20260926 05:55:36.802842 14217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:36.803164 14216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d8c4e1a5fef409f9ad62883f2b4f860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8c4e1a5fef409f9ad62883f2b4f860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45867 } } }
I20260926 05:55:36.803236 14216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860 [sys.catalog]: This master's current role is: LEADER
W20260926 05:55:36.803792 14230 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:55:36.803851 14230 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:55:36.803900 14224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:36.804059 14224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:36.805595 14224 catalog_manager.cc:1347] Generated new cluster ID: 277e7eae432e449e996498017000675f
I20260926 05:55:36.805651 14224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:36.808244 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:36.803549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:42653 (local address 127.0.0.1:45867)
0926 05:55:36.803743 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:36.803753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:36.803784 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:55:36.803910 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:36.803918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:36.803984 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:36.804131 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:36.804334 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:36.805419 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:36.806295 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:36.806780 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:36.806878 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:36.807576 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:36.807580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:36.807581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:36.807770 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:36.807801 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:36.807813 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:36.807885 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:36.807979 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 05:55:36.808008 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260926 05:55:36.809217 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:36.813464 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:36.810814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:57977 (local address 127.0.0.1:45867)
0926 05:55:36.810863 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:36.810865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:36.810872 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:55:36.810894 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:36.810895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:36.810905 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:36.810950 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:36.811094 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:36.811600 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:36.812288 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:36.812521 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:36.812575 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:36.813228 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:36.813230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:36.813230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:36.813235 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:36.813250 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:36.813255 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:36.813298 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:36.813365 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 05:55:36.813379 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:55:36.827013 14224 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:36.827590 14224 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:36.833590 14224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d8c4e1a5fef409f9ad62883f2b4f860: Generated new TSK 0
I20260926 05:55:36.833796 14224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:36.839442 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 14144
W20260926 05:55:36.843700 14129 connection.cc:535] client connection to 127.0.0.1:45867 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45867): Connection reset by peer (error 104)
2026-09-26T05:55:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:36 dist-test-slave-7fr8 krb5kdc[14243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:36 dist-test-slave-7fr8 krb5kdc[14243](info): set up 2 sockets
Sep 26 05:55:36 dist-test-slave-7fr8 krb5kdc[14243](info): commencing operation
krb5kdc: starting...
W20260926 05:55:37.818511 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:37 dist-test-slave-7fr8 krb5kdc[14243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:55:37.951481 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:38.048833 14256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:38.049151 14256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:38.049181 14256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:38.050750 14256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:38.050809 14256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:38.050823 14256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:38.050836 14256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:38.050848 14256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:38.052728 14256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:38.053045 14256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:38.053295 14256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:38.059938 14262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:38.059965 14261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:38.059903 14264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:38.060261 14256 server_base.cc:1034] running on GCE node
I20260926 05:55:38.060668 14256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:38.061058 14256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:38.062198 14256 hybrid_clock.cc:648] HybridClock initialized: now 1790402138062152 us; error 74 us; skew 500 ppm
Sep 26 05:55:38 dist-test-slave-7fr8 krb5kdc[14243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:38.065260 14256 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:55:38.065819 14256 webserver.cc:460] Webserver started at http://127.0.0.1:33329/ using document root <none> and password file <none>
I20260926 05:55:38.066083 14256 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:38.066174 14256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:38.066308 14256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:38.067595 14256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "ed0023d86c7f4017b298c2fc47f31700"
format_stamp: "Formatted at 2026-09-26 05:55:38 on dist-test-slave-7fr8"
server_key: "9692134e626fbae6aec592f3755b5b6e"
server_key_iv: "6d629a87172bf1c909a9770478c27cef"
server_key_version: "encryptionkey@0"
I20260926 05:55:38.067958 14256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "ed0023d86c7f4017b298c2fc47f31700"
format_stamp: "Formatted at 2026-09-26 05:55:38 on dist-test-slave-7fr8"
server_key: "9692134e626fbae6aec592f3755b5b6e"
server_key_iv: "6d629a87172bf1c909a9770478c27cef"
server_key_version: "encryptionkey@0"
I20260926 05:55:38.069680 14256 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 05:55:38.070923 14271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:38.071236 14256 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:38.071455 14256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "ed0023d86c7f4017b298c2fc47f31700"
format_stamp: "Formatted at 2026-09-26 05:55:38 on dist-test-slave-7fr8"
server_key: "9692134e626fbae6aec592f3755b5b6e"
server_key_iv: "6d629a87172bf1c909a9770478c27cef"
server_key_version: "encryptionkey@0"
I20260926 05:55:38.071569 14256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:38.075609 14256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:38.078445 14256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:38.078613 14256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:38.083050 14256 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38375
I20260926 05:55:38.083326 14323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38375 every 8 connection(s)
I20260926 05:55:38.083453 14256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:38.084599 14324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:38.087484 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 14256
I20260926 05:55:38.087586 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:38.087751 25596 external_mini_cluster.cc:1428] Setting key bcb83964484590cc84efb8d95f717144
I20260926 05:55:38.088044 14324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700: Bootstrap starting.
I20260926 05:55:38.088723 14324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:38.088985 14324 log.cc:824] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:38.089743 14324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700: No bootstrap required, opened a new log
I20260926 05:55:38.090444 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:38.088258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:50747 (local address 127.0.0.1:38375)
0926 05:55:38.088457 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:38.088466 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:38.088494 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:55:38.089102 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:38.089112 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:38.089154 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:38.089323 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:38.089988 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:38.089992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:38.089992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:38.089996 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:38.090034 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:38.090048 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:38.090178 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:38.090204 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 05:55:38.090246 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260926 05:55:38.091953 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:38.092247 14324 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0023d86c7f4017b298c2fc47f31700" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38375 } }
I20260926 05:55:38.092433 14324 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:38.092466 14324 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0023d86c7f4017b298c2fc47f31700, State: Initialized, Role: FOLLOWER
I20260926 05:55:38.092594 14324 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0023d86c7f4017b298c2fc47f31700" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38375 } }
I20260926 05:55:38.092697 14324 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:38.092790 14324 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:38.092850 14324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:38.093636 14324 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0023d86c7f4017b298c2fc47f31700" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38375 } }
I20260926 05:55:38.093746 14324 leader_election.cc:302] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed0023d86c7f4017b298c2fc47f31700; no voters:
I20260926 05:55:38.093923 14324 leader_election.cc:288] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:38.093959 14334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:38.094158 14334 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [term 1 LEADER]: Becoming Leader. State: Replica: ed0023d86c7f4017b298c2fc47f31700, State: Running, Role: LEADER
I20260926 05:55:38.094295 14334 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0023d86c7f4017b298c2fc47f31700" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38375 } }
I20260926 05:55:38.094302 14324 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:38.094831 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:38.093693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:44461 (local address 127.0.0.1:38375)
0926 05:55:38.093739 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:38.093741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:38.093824 (+ 83us) server_negotiation.cc:407] Connection header received
0926 05:55:38.093887 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:38.093888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:38.093894 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:38.093940 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:38.094634 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:38.094636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:38.094636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:38.094640 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:38.094655 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:38.094659 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:38.094706 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:38.094752 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 05:55:38.094766 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:55:38.095922 14335 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed0023d86c7f4017b298c2fc47f31700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0023d86c7f4017b298c2fc47f31700" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38375 } } }
I20260926 05:55:38.096007 14335 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [sys.catalog]: This master's current role is: LEADER
W20260926 05:55:38.096135 14349 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 05:55:38.096191 14336 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed0023d86c7f4017b298c2fc47f31700. Latest consensus state: current_term: 1 leader_uuid: "ed0023d86c7f4017b298c2fc47f31700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0023d86c7f4017b298c2fc47f31700" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38375 } } }
W20260926 05:55:38.096226 14349 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:55:38.096262 14336 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:38.096304 14351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:38.096464 14351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:38.098599 14351 catalog_manager.cc:1347] Generated new cluster ID: 668fa18bdfc645caa45efd074d1efd03
I20260926 05:55:38.098655 14351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:38.139768 14351 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:38.139976 14351 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:38.145965 14351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed0023d86c7f4017b298c2fc47f31700: Generated new TSK 0
I20260926 05:55:38.146307 14351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:38.157367 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 14256
2026-09-26T05:55:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:38 dist-test-slave-7fr8 krb5kdc[14355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:38 dist-test-slave-7fr8 krb5kdc[14355](info): set up 2 sockets
Sep 26 05:55:38 dist-test-slave-7fr8 krb5kdc[14355](info): commencing operation
krb5kdc: starting...
W20260926 05:55:39.151137 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:39 dist-test-slave-7fr8 krb5kdc[14355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:55:39.281852 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:39.365618 14368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:39.365851 14368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:39.365876 14368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:39.367105 14368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:39.367154 14368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:39.367168 14368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:39.367179 14368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:39.367192 14368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:39.369293 14368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:39.369565 14368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:39.369768 14368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:39.375375 14374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:39.375519 14368 server_base.cc:1034] running on GCE node
W20260926 05:55:39.375398 14373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:39.375412 14376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:39.375885 14368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:39.376189 14368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:39.377362 14368 hybrid_clock.cc:648] HybridClock initialized: now 1790402139377344 us; error 41 us; skew 500 ppm
Sep 26 05:55:39 dist-test-slave-7fr8 krb5kdc[14355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:39.385617 14368 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:55:39.386229 14368 webserver.cc:460] Webserver started at http://127.0.0.1:41361/ using document root <none> and password file <none>
I20260926 05:55:39.386482 14368 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:39.386538 14368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:39.386684 14368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:39.387751 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "e549184271484addb8b35b216624efef"
format_stamp: "Formatted at 2026-09-26 05:55:39 on dist-test-slave-7fr8"
server_key: "f74d65fbf146c7813dab5386c020c919"
server_key_iv: "3584d6c6d46b9e9a7cb35cb0b52cd60f"
server_key_version: "encryptionkey@0"
I20260926 05:55:39.388068 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "e549184271484addb8b35b216624efef"
format_stamp: "Formatted at 2026-09-26 05:55:39 on dist-test-slave-7fr8"
server_key: "f74d65fbf146c7813dab5386c020c919"
server_key_iv: "3584d6c6d46b9e9a7cb35cb0b52cd60f"
server_key_version: "encryptionkey@0"
I20260926 05:55:39.389467 14368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:55:39.390318 14383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:39.390551 14368 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:55:39.390643 14368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "e549184271484addb8b35b216624efef"
format_stamp: "Formatted at 2026-09-26 05:55:39 on dist-test-slave-7fr8"
server_key: "f74d65fbf146c7813dab5386c020c919"
server_key_iv: "3584d6c6d46b9e9a7cb35cb0b52cd60f"
server_key_version: "encryptionkey@0"
I20260926 05:55:39.390710 14368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:39.422422 14368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:39.424728 14368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:39.424885 14368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:39.428750 14368 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44061
I20260926 05:55:39.428781 14435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44061 every 8 connection(s)
I20260926 05:55:39.429105 14368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:39.429935 14436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:39.432226 14436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef: Bootstrap starting.
I20260926 05:55:39.432781 14436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:39.433030 14436 log.cc:824] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:39.433764 14436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef: No bootstrap required, opened a new log
I20260926 05:55:39.435460 14436 raft_consensus.cc:348] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e549184271484addb8b35b216624efef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } }
I20260926 05:55:39.435611 14436 raft_consensus.cc:374] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:39.435631 14436 raft_consensus.cc:729] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e549184271484addb8b35b216624efef, State: Initialized, Role: FOLLOWER
I20260926 05:55:39.435758 14436 consensus_queue.cc:260] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e549184271484addb8b35b216624efef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } }
I20260926 05:55:39.435824 14436 raft_consensus.cc:388] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:39.435848 14436 raft_consensus.cc:482] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:39.435896 14436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:39.436470 14436 raft_consensus.cc:504] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e549184271484addb8b35b216624efef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } }
I20260926 05:55:39.436559 14436 leader_election.cc:302] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e549184271484addb8b35b216624efef; no voters:
I20260926 05:55:39.436721 14436 leader_election.cc:288] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:39.436761 14439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:39.436923 14439 raft_consensus.cc:686] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [term 1 LEADER]: Becoming Leader. State: Replica: e549184271484addb8b35b216624efef, State: Running, Role: LEADER
I20260926 05:55:39.436988 14436 sys_catalog.cc:564] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:39.437014 14439 consensus_queue.cc:237] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e549184271484addb8b35b216624efef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } }
I20260926 05:55:39.437386 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 14368
I20260926 05:55:39.437505 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:39.437546 14440 sys_catalog.cc:455] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e549184271484addb8b35b216624efef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e549184271484addb8b35b216624efef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } } }
I20260926 05:55:39.437614 14440 sys_catalog.cc:458] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:39.437698 25596 external_mini_cluster.cc:1428] Setting key dd674fd1db6cedab178179acea0ae333
I20260926 05:55:39.437763 14441 sys_catalog.cc:455] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [sys.catalog]: SysCatalogTable state changed. Reason: New leader e549184271484addb8b35b216624efef. Latest consensus state: current_term: 1 leader_uuid: "e549184271484addb8b35b216624efef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e549184271484addb8b35b216624efef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } } }
I20260926 05:55:39.437812 14441 sys_catalog.cc:458] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:39.437996 14448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:39.438570 14448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:39.441416 14448 catalog_manager.cc:1347] Generated new cluster ID: 470cec8986aa428ebc7cfa75dc6a01b1
I20260926 05:55:39.441478 14448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:55:39 dist-test-slave-7fr8 krb5kdc[14355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 05:55:39.447342 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:39.438424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:51709 (local address 127.0.0.1:44061)
0926 05:55:39.438781 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:39.438791 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:39.438819 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:55:39.438948 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:39.438955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:39.439016 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:39.439156 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:39.439869 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:39.440747 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:39.441589 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:39.442002 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:39.442120 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:39.444314 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:39.444318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:39.444318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:39.444331 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:39.445820 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:39.446197 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:39.446200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:39.446201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:39.446300 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:39.446512 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:39.446513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:39.446514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:39.446761 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:39.446909 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:39.447028 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 05:55:39.447140 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":67,"threads_started":1}
I20260926 05:55:39.448356 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:39.460991 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:39.454461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:59737 (local address 127.0.0.1:44061)
0926 05:55:39.454526 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:39.454529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:39.454660 (+ 131us) server_negotiation.cc:407] Connection header received
0926 05:55:39.454724 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:39.454725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:39.454735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:39.454789 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:39.455330 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:39.455845 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:39.456522 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:39.456737 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:39.456792 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:39.458540 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:39.458541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:39.458542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:39.458548 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:39.459770 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:39.460224 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:39.460225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:39.460226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:39.460287 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:39.460493 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:39.460493 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:39.460494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:39.460608 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:39.460699 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:39.460835 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 05:55:39.460898 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:55:39.473388 14448 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:39.473992 14448 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:39.480202 14448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e549184271484addb8b35b216624efef: Generated new TSK 0
I20260926 05:55:39.480408 14448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:39.484067 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 14368
W20260926 05:55:39.487951 14353 connection.cc:535] client connection to 127.0.0.1:44061 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44061): Connection reset by peer (error 104)
2026-09-26T05:55:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:39 dist-test-slave-7fr8 krb5kdc[14467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:39 dist-test-slave-7fr8 krb5kdc[14467](info): set up 2 sockets
Sep 26 05:55:39 dist-test-slave-7fr8 krb5kdc[14467](info): commencing operation
krb5kdc: starting...
W20260926 05:55:40.462393 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:40 dist-test-slave-7fr8 krb5kdc[14467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:55:40.589942 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:40.673207 14480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:40.673446 14480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:40.673481 14480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:40.675166 14480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:40.675243 14480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:40.675267 14480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:40.675285 14480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:40.675299 14480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:40.677049 14480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37921
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:40.677342 14480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:40.677577 14480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:40.682443 14486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:40.682446 14485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:40.682554 14488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:40.682760 14480 server_base.cc:1034] running on GCE node
I20260926 05:55:40.682986 14480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:40.683279 14480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:40.684453 14480 hybrid_clock.cc:648] HybridClock initialized: now 1790402140684440 us; error 29 us; skew 500 ppm
Sep 26 05:55:40 dist-test-slave-7fr8 krb5kdc[14467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:40.687467 14480 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:55:40.687971 14480 webserver.cc:460] Webserver started at http://127.0.0.1:33795/ using document root <none> and password file <none>
I20260926 05:55:40.688198 14480 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:40.688241 14480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:40.688329 14480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:40.689363 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "27917e7f572c4eb290b5d0958ad79bba"
format_stamp: "Formatted at 2026-09-26 05:55:40 on dist-test-slave-7fr8"
server_key: "0055194112370f8a02cd430a2f7be70d"
server_key_iv: "8b19ea9bdf6c42c81467dd199bfa0d37"
server_key_version: "encryptionkey@0"
I20260926 05:55:40.689668 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "27917e7f572c4eb290b5d0958ad79bba"
format_stamp: "Formatted at 2026-09-26 05:55:40 on dist-test-slave-7fr8"
server_key: "0055194112370f8a02cd430a2f7be70d"
server_key_iv: "8b19ea9bdf6c42c81467dd199bfa0d37"
server_key_version: "encryptionkey@0"
I20260926 05:55:40.691116 14480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:55:40.692148 14495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:40.692346 14480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:55:40.692426 14480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "27917e7f572c4eb290b5d0958ad79bba"
format_stamp: "Formatted at 2026-09-26 05:55:40 on dist-test-slave-7fr8"
server_key: "0055194112370f8a02cd430a2f7be70d"
server_key_iv: "8b19ea9bdf6c42c81467dd199bfa0d37"
server_key_version: "encryptionkey@0"
I20260926 05:55:40.692490 14480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:40.713142 14480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:40.716282 14480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:40.716467 14480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:40.720749 14480 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37921
I20260926 05:55:40.720842 14547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37921 every 8 connection(s)
I20260926 05:55:40.721197 14480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:40.722229 14548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:40.724628 14548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba: Bootstrap starting.
I20260926 05:55:40.725265 14548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:40.725538 14548 log.cc:824] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:40.725760 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 14480
I20260926 05:55:40.725857 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:40.726042 25596 external_mini_cluster.cc:1428] Setting key 2a7f336b381d25a028e769200551cd27
I20260926 05:55:40.726363 14548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba: No bootstrap required, opened a new log
I20260926 05:55:40.730017 14548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27917e7f572c4eb290b5d0958ad79bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37921 } }
I20260926 05:55:40.730283 14548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:40.730324 14548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27917e7f572c4eb290b5d0958ad79bba, State: Initialized, Role: FOLLOWER
I20260926 05:55:40.730470 14548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27917e7f572c4eb290b5d0958ad79bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37921 } }
I20260926 05:55:40.730551 14548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:40.730583 14548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:40.730638 14548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:40.731367 14548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27917e7f572c4eb290b5d0958ad79bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37921 } }
I20260926 05:55:40.731484 14548 leader_election.cc:302] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27917e7f572c4eb290b5d0958ad79bba; no voters:
I20260926 05:55:40.731660 14548 leader_election.cc:288] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:40.731853 14553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:40.731945 14548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:40.732303 14553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [term 1 LEADER]: Becoming Leader. State: Replica: 27917e7f572c4eb290b5d0958ad79bba, State: Running, Role: LEADER
I20260926 05:55:40.732499 14553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27917e7f572c4eb290b5d0958ad79bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37921 } }
I20260926 05:55:40.733183 14555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27917e7f572c4eb290b5d0958ad79bba. Latest consensus state: current_term: 1 leader_uuid: "27917e7f572c4eb290b5d0958ad79bba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27917e7f572c4eb290b5d0958ad79bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37921 } } }
I20260926 05:55:40.733307 14554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27917e7f572c4eb290b5d0958ad79bba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27917e7f572c4eb290b5d0958ad79bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37921 } } }
I20260926 05:55:40.733392 14554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:40.733381 14555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:40.733632 14565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:40.734221 14565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:40.734283 14552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:40.726686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:49735 (local address 127.0.0.1:37921)
0926 05:55:40.726888 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:40.726898 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:40.726934 (+ 36us) server_negotiation.cc:407] Connection header received
0926 05:55:40.727714 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:40.727725 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:40.727801 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:40.727977 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:40.728444 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:40.730218 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:40.731191 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:40.731835 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:40.732565 (+ 730us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:40.733297 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:40.733302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:40.733302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:40.733523 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:40.733636 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:40.733657 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:40.733740 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:40.734016 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 05:55:40.734048 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260926 05:55:40.735601 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:40.738070 14565 catalog_manager.cc:1347] Generated new cluster ID: ad07e974ceb44deabe824c5a1f40215c
I20260926 05:55:40.738169 14565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:40.741070 14552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:40.737000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:37443 (local address 127.0.0.1:37921)
0926 05:55:40.737038 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:40.737040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:40.737119 (+ 79us) server_negotiation.cc:407] Connection header received
0926 05:55:40.737184 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:40.737185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:40.737196 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:40.737257 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:40.737401 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:40.738232 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:40.739362 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:40.739697 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:40.739774 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:40.740684 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:40.740686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:40.740687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:40.740693 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:40.740711 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:40.740716 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:40.740784 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:40.740968 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 05:55:40.740980 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:55:40.754870 14565 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:40.755406 14565 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:40.761618 14565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27917e7f572c4eb290b5d0958ad79bba: Generated new TSK 0
I20260926 05:55:40.761801 14565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:40.768265 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 14480
W20260926 05:55:40.772435 14465 connection.cc:535] client connection to 127.0.0.1:37921 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37921): Connection reset by peer (error 104)
2026-09-26T05:55:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1284 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:40 dist-test-slave-7fr8 krb5kdc[14579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:40 dist-test-slave-7fr8 krb5kdc[14579](info): set up 2 sockets
Sep 26 05:55:40 dist-test-slave-7fr8 krb5kdc[14579](info): commencing operation
krb5kdc: starting...
W20260926 05:55:41.734411 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:41 dist-test-slave-7fr8 krb5kdc[14579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:55:41.865338 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:41.950840 14592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:41.951077 14592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:41.951112 14592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:41.952359 14592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:41.952414 14592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:41.952431 14592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:41.952448 14592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:41.952463 14592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:41.954154 14592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:41.954427 14592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:41.954658 14592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:41.959436 14598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:41.959424 14597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:41.959551 14600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:41.960448 14592 server_base.cc:1034] running on GCE node
I20260926 05:55:41.960731 14592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:41.961036 14592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:41.962185 14592 hybrid_clock.cc:648] HybridClock initialized: now 1790402141962149 us; error 50 us; skew 500 ppm
Sep 26 05:55:41 dist-test-slave-7fr8 krb5kdc[14579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:41.964890 14592 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:55:41.965333 14592 webserver.cc:460] Webserver started at http://127.0.0.1:40477/ using document root <none> and password file <none>
I20260926 05:55:41.965517 14592 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:41.965557 14592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:41.965651 14592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:41.966795 14592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "3fd7c797bf8948fcbf5017001440524d"
format_stamp: "Formatted at 2026-09-26 05:55:41 on dist-test-slave-7fr8"
server_key: "4dcef5f7db2af921d97de5745c090629"
server_key_iv: "6633f9ef44e979f82ddd24f972d485aa"
server_key_version: "encryptionkey@0"
I20260926 05:55:41.967110 14592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "3fd7c797bf8948fcbf5017001440524d"
format_stamp: "Formatted at 2026-09-26 05:55:41 on dist-test-slave-7fr8"
server_key: "4dcef5f7db2af921d97de5745c090629"
server_key_iv: "6633f9ef44e979f82ddd24f972d485aa"
server_key_version: "encryptionkey@0"
I20260926 05:55:41.968400 14592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:55:41.969151 14607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:41.969301 14592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:41.969383 14592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "3fd7c797bf8948fcbf5017001440524d"
format_stamp: "Formatted at 2026-09-26 05:55:41 on dist-test-slave-7fr8"
server_key: "4dcef5f7db2af921d97de5745c090629"
server_key_iv: "6633f9ef44e979f82ddd24f972d485aa"
server_key_version: "encryptionkey@0"
I20260926 05:55:41.969455 14592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:41.972924 14592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:41.975914 14592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:41.976058 14592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:41.980106 14592 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39281
I20260926 05:55:41.980185 14659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39281 every 8 connection(s)
I20260926 05:55:41.980520 14592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:41.981564 14660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:41.984054 14660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d: Bootstrap starting.
I20260926 05:55:41.984632 14660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:41.984889 14660 log.cc:824] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:41.985586 14660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d: No bootstrap required, opened a new log
I20260926 05:55:41.987365 14660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd7c797bf8948fcbf5017001440524d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39281 } }
I20260926 05:55:41.987521 14660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:41.987553 14660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd7c797bf8948fcbf5017001440524d, State: Initialized, Role: FOLLOWER
I20260926 05:55:41.987689 14660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd7c797bf8948fcbf5017001440524d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39281 } }
I20260926 05:55:41.987767 14660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:41.987800 14660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:41.987850 14660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:41.988463 14660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd7c797bf8948fcbf5017001440524d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39281 } }
I20260926 05:55:41.988562 14660 leader_election.cc:302] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fd7c797bf8948fcbf5017001440524d; no voters:
I20260926 05:55:41.988744 14660 leader_election.cc:288] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:41.988853 14663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:41.989008 14660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:41.989017 14663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [term 1 LEADER]: Becoming Leader. State: Replica: 3fd7c797bf8948fcbf5017001440524d, State: Running, Role: LEADER
I20260926 05:55:41.989146 14663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd7c797bf8948fcbf5017001440524d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39281 } }
I20260926 05:55:41.989497 14664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fd7c797bf8948fcbf5017001440524d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd7c797bf8948fcbf5017001440524d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39281 } } }
I20260926 05:55:41.989476 14665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fd7c797bf8948fcbf5017001440524d. Latest consensus state: current_term: 1 leader_uuid: "3fd7c797bf8948fcbf5017001440524d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd7c797bf8948fcbf5017001440524d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39281 } } }
I20260926 05:55:41.989571 14664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:41.989578 14665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:41.989796 14667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:41.990370 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 14592
I20260926 05:55:41.990471 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:41.990687 25596 external_mini_cluster.cc:1428] Setting key 67e4dfddf100d30bf357cf5e76232c03
I20260926 05:55:41.991271 14667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:41.993405 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:41.991450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:37175 (local address 127.0.0.1:39281)
0926 05:55:41.991671 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:41.991681 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:41.991897 (+ 216us) server_negotiation.cc:407] Connection header received
0926 05:55:41.992040 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:41.992048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:41.992080 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:41.992233 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:41.992973 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:41.992976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:41.992976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:41.992981 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:41.993023 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:41.993035 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:41.993092 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:41.993167 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 05:55:41.993205 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260926 05:55:41.994382 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:41.994580 14667 catalog_manager.cc:1347] Generated new cluster ID: 68bd8a19a3fd4c1895d450d974bd30b7
I20260926 05:55:41.994627 14667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:41.997356 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:41.996155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.7:54469 (local address 127.0.0.1:39281)
0926 05:55:41.996288 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:41.996290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:41.996416 (+ 126us) server_negotiation.cc:407] Connection header received
0926 05:55:41.996483 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:41.996483 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:41.996489 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:41.996544 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:41.997135 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:41.997136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:41.997136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:41.997140 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:41.997156 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:41.997160 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:41.997209 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:41.997277 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 05:55:41.997290 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 05:55:42.027210 14667 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:42.027420 14667 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:42.036837 14667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fd7c797bf8948fcbf5017001440524d: Generated new TSK 0
I20260926 05:55:42.037099 14667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:42.039144 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 14592
2026-09-26T05:55:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:42 dist-test-slave-7fr8 krb5kdc[14691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:42 dist-test-slave-7fr8 krb5kdc[14691](info): set up 2 sockets
Sep 26 05:55:42 dist-test-slave-7fr8 krb5kdc[14691](info): commencing operation
krb5kdc: starting...
W20260926 05:55:43.010474 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:43 dist-test-slave-7fr8 krb5kdc[14691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:55:43.131021 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:46433
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:46433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:43.212808 14704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:43.213088 14704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:43.213114 14704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:43.214406 14704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:43.214455 14704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:43.214468 14704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:43.214478 14704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:43.214490 14704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:43.216100 14704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:46433
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:46433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:43.216351 14704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:43.216559 14704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:43.221334 14710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:43.221508 14704 server_base.cc:1034] running on GCE node
W20260926 05:55:43.221329 14712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:43.221418 14709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:43.221844 14704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:43.222182 14704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:43.223343 14704 hybrid_clock.cc:648] HybridClock initialized: now 1790402143223326 us; error 29 us; skew 500 ppm
Sep 26 05:55:43 dist-test-slave-7fr8 krb5kdc[14691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402143, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:43.225934 14704 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:55:43.226438 14704 webserver.cc:460] Webserver started at http://127.24.255.62:41467/ using document root <none> and password file <none>
I20260926 05:55:43.226768 14704 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:43.226845 14704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:43.226991 14704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:43.228474 14704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "ab9db623bc084f3a8c3d431abe02679b"
format_stamp: "Formatted at 2026-09-26 05:55:43 on dist-test-slave-7fr8"
server_key: "dafffe5e211755727b21e47e1de9a2de"
server_key_iv: "edb2288f78417b14916a450fc1d4b78a"
server_key_version: "encryptionkey@0"
I20260926 05:55:43.228844 14704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "ab9db623bc084f3a8c3d431abe02679b"
format_stamp: "Formatted at 2026-09-26 05:55:43 on dist-test-slave-7fr8"
server_key: "dafffe5e211755727b21e47e1de9a2de"
server_key_iv: "edb2288f78417b14916a450fc1d4b78a"
server_key_version: "encryptionkey@0"
I20260926 05:55:43.230297 14704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:55:43.231177 14719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:43.231371 14704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:43.231458 14704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "ab9db623bc084f3a8c3d431abe02679b"
format_stamp: "Formatted at 2026-09-26 05:55:43 on dist-test-slave-7fr8"
server_key: "dafffe5e211755727b21e47e1de9a2de"
server_key_iv: "edb2288f78417b14916a450fc1d4b78a"
server_key_version: "encryptionkey@0"
I20260926 05:55:43.231527 14704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:43.252758 14704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:43.255448 14704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:43.255637 14704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:43.259466 14704 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:46433
I20260926 05:55:43.259557 14771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:46433 every 8 connection(s)
I20260926 05:55:43.259879 14704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:43.260819 14772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:43.263140 14772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b: Bootstrap starting.
I20260926 05:55:43.263736 14772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:43.263990 14772 log.cc:824] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:43.264746 14772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b: No bootstrap required, opened a new log
I20260926 05:55:43.266242 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 14704
I20260926 05:55:43.266353 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:43.266531 25596 external_mini_cluster.cc:1428] Setting key f0d5d4740b3d7f58510bce5437c388f4
I20260926 05:55:43.266449 14772 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9db623bc084f3a8c3d431abe02679b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46433 } }
I20260926 05:55:43.266613 14772 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:43.266654 14772 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9db623bc084f3a8c3d431abe02679b, State: Initialized, Role: FOLLOWER
I20260926 05:55:43.266808 14772 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9db623bc084f3a8c3d431abe02679b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46433 } }
I20260926 05:55:43.266894 14772 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:43.266933 14772 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:43.266976 14772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:43.268020 14772 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9db623bc084f3a8c3d431abe02679b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46433 } }
I20260926 05:55:43.268163 14772 leader_election.cc:302] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab9db623bc084f3a8c3d431abe02679b; no voters:
I20260926 05:55:43.268338 14772 leader_election.cc:288] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:43.268371 14777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:43.268514 14777 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [term 1 LEADER]: Becoming Leader. State: Replica: ab9db623bc084f3a8c3d431abe02679b, State: Running, Role: LEADER
I20260926 05:55:43.268672 14772 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:43.268895 14777 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9db623bc084f3a8c3d431abe02679b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46433 } }
I20260926 05:55:43.269299 14778 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab9db623bc084f3a8c3d431abe02679b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9db623bc084f3a8c3d431abe02679b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46433 } } }
I20260926 05:55:43.269497 14778 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:43.269438 14779 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9db623bc084f3a8c3d431abe02679b. Latest consensus state: current_term: 1 leader_uuid: "ab9db623bc084f3a8c3d431abe02679b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9db623bc084f3a8c3d431abe02679b" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46433 } } }
I20260926 05:55:43.269719 14779 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b [sys.catalog]: This master's current role is: LEADER
W20260926 05:55:43.270658 14793 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:55:43.270716 14793 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:55:43.270769 14786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:43.270944 14786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:43.272682 14786 catalog_manager.cc:1347] Generated new cluster ID: 2664602d4cf845edb919d7fa31b92c6e
I20260926 05:55:43.272734 14786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:55:43 dist-test-slave-7fr8 krb5kdc[14691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:43.277264 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:43.267137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51251 (local address 127.24.255.62:46433)
0926 05:55:43.267321 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:43.267332 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:43.267359 (+ 27us) server_negotiation.cc:407] Connection header received
0926 05:55:43.267462 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:43.267470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:43.267513 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:43.267631 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:43.268424 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:43.269944 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:43.271055 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:43.271519 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:43.271629 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:43.274024 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:43.274028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:43.274028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:43.274040 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:43.275692 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:43.276088 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:43.276091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:43.276092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:43.276193 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:43.276400 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:43.276401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:43.276401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:43.276677 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:43.276840 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:43.276974 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 05:55:43.277083 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260926 05:55:43.277944 14786 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:43.278282 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:43.278654 14786 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:43.284233 14786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab9db623bc084f3a8c3d431abe02679b: Generated new TSK 0
I20260926 05:55:43.284471 14786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:43.291522 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:43.282596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52321 (local address 127.24.255.62:46433)
0926 05:55:43.282640 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:43.282642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:43.284717 (+ 2075us) server_negotiation.cc:407] Connection header received
0926 05:55:43.284755 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:43.284756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:43.284767 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:43.284840 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:43.285386 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:43.286307 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:43.287106 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:43.287451 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:43.287519 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:43.288771 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:43.288772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:43.288773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:43.288780 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:43.290270 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:43.290688 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:43.290689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:43.290689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:43.290754 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:43.290941 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:43.290942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:43.290942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:43.291089 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:43.291196 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:43.291353 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 05:55:43.291419 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:55:43.293483 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 14704
W20260926 05:55:43.297497 14689 connection.cc:535] client connection to 127.24.255.62:46433 recv error: Network error: failed to read from TLS socket (remote: 127.24.255.62:46433): Connection reset by peer (error 104)
2026-09-26T05:55:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:43 dist-test-slave-7fr8 krb5kdc[14803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:43 dist-test-slave-7fr8 krb5kdc[14803](info): set up 2 sockets
Sep 26 05:55:43 dist-test-slave-7fr8 krb5kdc[14803](info): commencing operation
krb5kdc: starting...
W20260926 05:55:44.258415 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:44 dist-test-slave-7fr8 krb5kdc[14803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:55:44.381152 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:42035
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:46865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:42035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:44.464440 14816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:44.464681 14816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:44.464705 14816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:44.465895 14816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:44.465935 14816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:44.465948 14816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:44.465960 14816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:44.465970 14816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:44.467535 14816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:46865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:42035
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:42035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:44.467757 14816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:44.467941 14816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:44.472844 14821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:44.472891 14824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:44.472847 14816 server_base.cc:1034] running on GCE node
W20260926 05:55:44.472843 14822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:44.473291 14816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:44.473631 14816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:44.474782 14816 hybrid_clock.cc:648] HybridClock initialized: now 1790402144474752 us; error 46 us; skew 500 ppm
Sep 26 05:55:44 dist-test-slave-7fr8 krb5kdc[14803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402144, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:44.482864 14816 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:55:44.483366 14816 webserver.cc:460] Webserver started at http://127.24.255.62:37521/ using document root <none> and password file <none>
I20260926 05:55:44.483597 14816 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:44.483646 14816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:44.483758 14816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:44.484853 14816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "753707b778364e56b8da6b8ebc5adbc8"
format_stamp: "Formatted at 2026-09-26 05:55:44 on dist-test-slave-7fr8"
server_key: "d544ee24021eb696fb20b2818e861f20"
server_key_iv: "c16437182b4e2a6ea2b2a06a41218be6"
server_key_version: "encryptionkey@0"
I20260926 05:55:44.485186 14816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "753707b778364e56b8da6b8ebc5adbc8"
format_stamp: "Formatted at 2026-09-26 05:55:44 on dist-test-slave-7fr8"
server_key: "d544ee24021eb696fb20b2818e861f20"
server_key_iv: "c16437182b4e2a6ea2b2a06a41218be6"
server_key_version: "encryptionkey@0"
I20260926 05:55:44.486680 14816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:55:44.487578 14831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:44.487807 14816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:55:44.487901 14816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "753707b778364e56b8da6b8ebc5adbc8"
format_stamp: "Formatted at 2026-09-26 05:55:44 on dist-test-slave-7fr8"
server_key: "d544ee24021eb696fb20b2818e861f20"
server_key_iv: "c16437182b4e2a6ea2b2a06a41218be6"
server_key_version: "encryptionkey@0"
I20260926 05:55:44.487972 14816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:44.521054 14816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:44.524376 14816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:44.524633 14816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:44.528945 14816 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:42035
I20260926 05:55:44.529206 14883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:42035 every 8 connection(s)
I20260926 05:55:44.529343 14816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:44.530083 14884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:44.532536 14884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8: Bootstrap starting.
I20260926 05:55:44.533113 14884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:44.533354 14884 log.cc:824] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:44.534083 14884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8: No bootstrap required, opened a new log
I20260926 05:55:44.535990 14884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753707b778364e56b8da6b8ebc5adbc8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42035 } }
I20260926 05:55:44.536158 14884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:44.536181 14884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753707b778364e56b8da6b8ebc5adbc8, State: Initialized, Role: FOLLOWER
I20260926 05:55:44.536296 14884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753707b778364e56b8da6b8ebc5adbc8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42035 } }
I20260926 05:55:44.536348 14884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:44.536393 14884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:44.536448 14884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:44.536882 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 14816
I20260926 05:55:44.536990 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:44.537168 25596 external_mini_cluster.cc:1428] Setting key ff6ec40e28349cbcd10a98aba4ac350a
I20260926 05:55:44.537544 14884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753707b778364e56b8da6b8ebc5adbc8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42035 } }
I20260926 05:55:44.537740 14884 leader_election.cc:302] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 753707b778364e56b8da6b8ebc5adbc8; no voters:
I20260926 05:55:44.537925 14884 leader_election.cc:288] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:44.537946 14889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:44.538056 14889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [term 1 LEADER]: Becoming Leader. State: Replica: 753707b778364e56b8da6b8ebc5adbc8, State: Running, Role: LEADER
I20260926 05:55:44.538221 14889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753707b778364e56b8da6b8ebc5adbc8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42035 } }
I20260926 05:55:44.538223 14884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:44.538650 14891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 753707b778364e56b8da6b8ebc5adbc8. Latest consensus state: current_term: 1 leader_uuid: "753707b778364e56b8da6b8ebc5adbc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753707b778364e56b8da6b8ebc5adbc8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42035 } } }
I20260926 05:55:44.538820 14891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:44.539006 14890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "753707b778364e56b8da6b8ebc5adbc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753707b778364e56b8da6b8ebc5adbc8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42035 } } }
I20260926 05:55:44.539124 14890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:44.539351 14893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:44.539873 14893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:44.541484 14893 catalog_manager.cc:1347] Generated new cluster ID: 1c46e99c44244ebf80be952121a8a241
I20260926 05:55:44.541545 14893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:44.543746 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:44.537692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35021 (local address 127.24.255.62:42035)
0926 05:55:44.537940 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:44.537948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:44.537981 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:55:44.538165 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:44.538173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:44.538236 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:44.538389 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:44.538599 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:44.539962 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:44.541086 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:44.541700 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:44.541822 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:44.543029 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:44.543034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:44.543034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:44.543263 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:44.543305 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:44.543321 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:44.543408 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:44.543492 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 05:55:44.543524 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
I20260926 05:55:44.544726 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:44.549796 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:44.546228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38673 (local address 127.24.255.62:42035)
0926 05:55:44.546264 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:44.546265 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:44.546408 (+ 143us) server_negotiation.cc:407] Connection header received
0926 05:55:44.546457 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:44.546458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:44.546470 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:44.546535 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:44.546661 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:44.547415 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:44.548418 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:44.548754 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:44.548827 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:44.549536 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:44.549538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:44.549538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:44.549544 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:44.549562 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:44.549567 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:44.549625 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:44.549690 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 05:55:44.549705 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:55:44.558204 14893 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:44.558992 14893 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:44.566752 14893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 753707b778364e56b8da6b8ebc5adbc8: Generated new TSK 0
I20260926 05:55:44.566998 14893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:44.570585 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 14816
W20260926 05:55:44.575441 14801 connection.cc:535] client connection to 127.24.255.62:42035 recv error: Network error: failed to read from TLS socket (remote: 127.24.255.62:42035): Connection reset by peer (error 104)
2026-09-26T05:55:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:44 dist-test-slave-7fr8 krb5kdc[14915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:44 dist-test-slave-7fr8 krb5kdc[14915](info): set up 2 sockets
Sep 26 05:55:44 dist-test-slave-7fr8 krb5kdc[14915](info): commencing operation
krb5kdc: starting...
W20260926 05:55:45.538337 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:45 dist-test-slave-7fr8 krb5kdc[14915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:55:45.662730 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:46671
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:46671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:45.743538 14928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:45.743752 14928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:45.743774 14928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:45.744963 14928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:45.745014 14928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:45.745028 14928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:45.745038 14928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:45.745049 14928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:45.746634 14928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:46671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:46671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:45.746868 14928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:45.747079 14928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:45.751896 14936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:45.751945 14928 server_base.cc:1034] running on GCE node
W20260926 05:55:45.751955 14934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:45.751904 14933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:45.752394 14928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:45.752696 14928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:45.753855 14928 hybrid_clock.cc:648] HybridClock initialized: now 1790402145753834 us; error 38 us; skew 500 ppm
Sep 26 05:55:45 dist-test-slave-7fr8 krb5kdc[14915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402145, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:45.756346 14928 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:55:45.756837 14928 webserver.cc:460] Webserver started at http://127.24.255.62:44305/ using document root <none> and password file <none>
I20260926 05:55:45.757067 14928 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:45.757117 14928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:45.757236 14928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:45.758335 14928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "30c84922e6e54b3e9e2c3740add4cdbd"
format_stamp: "Formatted at 2026-09-26 05:55:45 on dist-test-slave-7fr8"
server_key: "5d44a65ec4a706bc08201848b144d1eb"
server_key_iv: "23b8b6250089a06af3a1957e76610827"
server_key_version: "encryptionkey@0"
I20260926 05:55:45.758666 14928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "30c84922e6e54b3e9e2c3740add4cdbd"
format_stamp: "Formatted at 2026-09-26 05:55:45 on dist-test-slave-7fr8"
server_key: "5d44a65ec4a706bc08201848b144d1eb"
server_key_iv: "23b8b6250089a06af3a1957e76610827"
server_key_version: "encryptionkey@0"
I20260926 05:55:45.759915 14928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:55:45.760722 14943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:45.760876 14928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:45.760960 14928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "30c84922e6e54b3e9e2c3740add4cdbd"
format_stamp: "Formatted at 2026-09-26 05:55:45 on dist-test-slave-7fr8"
server_key: "5d44a65ec4a706bc08201848b144d1eb"
server_key_iv: "23b8b6250089a06af3a1957e76610827"
server_key_version: "encryptionkey@0"
I20260926 05:55:45.761022 14928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:45.764369 14928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:45.766409 14928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:45.766795 14928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:45.770663 14928 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:46671
I20260926 05:55:45.770741 14995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:46671 every 8 connection(s)
I20260926 05:55:45.771041 14928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:45.772051 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:45.774482 14996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd: Bootstrap starting.
I20260926 05:55:45.775069 14996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:45.775336 14996 log.cc:824] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:45.775996 14996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd: No bootstrap required, opened a new log
I20260926 05:55:45.777601 14996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c84922e6e54b3e9e2c3740add4cdbd" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46671 } }
I20260926 05:55:45.777777 14996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:45.777815 14996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c84922e6e54b3e9e2c3740add4cdbd, State: Initialized, Role: FOLLOWER
I20260926 05:55:45.777850 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 14928
I20260926 05:55:45.777958 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:45.777973 14996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c84922e6e54b3e9e2c3740add4cdbd" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46671 } }
I20260926 05:55:45.778064 14996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:45.778134 14996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:45.778187 25596 external_mini_cluster.cc:1428] Setting key 776e8c74ee8d2c96220a32629b6efbc1
I20260926 05:55:45.778187 14996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:45.779160 14996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c84922e6e54b3e9e2c3740add4cdbd" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46671 } }
I20260926 05:55:45.779271 14996 leader_election.cc:302] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30c84922e6e54b3e9e2c3740add4cdbd; no voters:
I20260926 05:55:45.779429 14996 leader_election.cc:288] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:45.779471 15001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:45.779606 15001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [term 1 LEADER]: Becoming Leader. State: Replica: 30c84922e6e54b3e9e2c3740add4cdbd, State: Running, Role: LEADER
I20260926 05:55:45.779690 14996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:45.779690 15001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c84922e6e54b3e9e2c3740add4cdbd" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46671 } }
I20260926 05:55:45.780082 15003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30c84922e6e54b3e9e2c3740add4cdbd. Latest consensus state: current_term: 1 leader_uuid: "30c84922e6e54b3e9e2c3740add4cdbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c84922e6e54b3e9e2c3740add4cdbd" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46671 } } }
I20260926 05:55:45.780161 15003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:45.780230 15002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30c84922e6e54b3e9e2c3740add4cdbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c84922e6e54b3e9e2c3740add4cdbd" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 46671 } } }
I20260926 05:55:45.780782 15002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:45.780624 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:45.778751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43831 (local address 127.24.255.62:46671)
0926 05:55:45.778922 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:45.778931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:45.778968 (+ 37us) server_negotiation.cc:407] Connection header received
0926 05:55:45.779074 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:45.779080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:45.779103 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:45.779209 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:45.779925 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:45.779929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:45.779929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:45.779934 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:45.779983 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:45.779998 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:45.780062 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:45.780389 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 05:55:45.780446 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260926 05:55:45.780385 15008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:45.781319 15008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:45.781999 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:45.783486 15008 catalog_manager.cc:1347] Generated new cluster ID: 83fc361d59ad4baf835db06b7d759b3d
I20260926 05:55:45.783545 15008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:45.790519 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:45.789239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55131 (local address 127.24.255.62:46671)
0926 05:55:45.789327 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:45.789329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:45.789344 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:55:45.789377 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:45.789378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:45.789384 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:45.789471 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:45.790268 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:45.790270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:45.790271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:45.790276 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:45.790296 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:45.790300 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:45.790361 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:45.790412 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 05:55:45.790425 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 05:55:45.806536 15008 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:45.806746 15008 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:45.810055 15008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30c84922e6e54b3e9e2c3740add4cdbd: Generated new TSK 0
I20260926 05:55:45.810369 15008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:45.816154 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 14928
2026-09-26T05:55:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1244 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:45 dist-test-slave-7fr8 krb5kdc[15027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:45 dist-test-slave-7fr8 krb5kdc[15027](info): set up 2 sockets
Sep 26 05:55:45 dist-test-slave-7fr8 krb5kdc[15027](info): commencing operation
krb5kdc: starting...
W20260926 05:55:46.774276 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:46 dist-test-slave-7fr8 krb5kdc[15027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:55:46.897743 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:33915
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:33915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:46.982528 15040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:46.982771 15040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:46.982796 15040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:46.984038 15040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:46.984099 15040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:46.984113 15040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:46.984125 15040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:46.984136 15040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:46.985752 15040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:33915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:46.986022 15040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:46.986266 15040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:46.991402 15045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:46.991407 15046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:46.991458 15048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:46.991714 15040 server_base.cc:1034] running on GCE node
I20260926 05:55:46.992283 15040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:46.993064 15040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:46.994410 15040 hybrid_clock.cc:648] HybridClock initialized: now 1790402146994359 us; error 68 us; skew 500 ppm
Sep 26 05:55:46 dist-test-slave-7fr8 krb5kdc[15027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402146, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:46.997174 15040 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:55:46.997789 15040 webserver.cc:460] Webserver started at http://127.24.255.62:35417/ using document root <none> and password file <none>
I20260926 05:55:46.998010 15040 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:46.998065 15040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:46.998209 15040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:46.999310 15040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "a93e1713301846d69c9d2e2aad7d3f28"
format_stamp: "Formatted at 2026-09-26 05:55:46 on dist-test-slave-7fr8"
server_key: "5140e391a9f8fb0d87fe2eabf504876b"
server_key_iv: "7a18f3b181a3ec48bf8e74f426179fb5"
server_key_version: "encryptionkey@0"
I20260926 05:55:46.999651 15040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "a93e1713301846d69c9d2e2aad7d3f28"
format_stamp: "Formatted at 2026-09-26 05:55:46 on dist-test-slave-7fr8"
server_key: "5140e391a9f8fb0d87fe2eabf504876b"
server_key_iv: "7a18f3b181a3ec48bf8e74f426179fb5"
server_key_version: "encryptionkey@0"
I20260926 05:55:47.001138 15040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:55:47.002243 15055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:47.002446 15040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:47.002542 15040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "a93e1713301846d69c9d2e2aad7d3f28"
format_stamp: "Formatted at 2026-09-26 05:55:46 on dist-test-slave-7fr8"
server_key: "5140e391a9f8fb0d87fe2eabf504876b"
server_key_iv: "7a18f3b181a3ec48bf8e74f426179fb5"
server_key_version: "encryptionkey@0"
I20260926 05:55:47.002612 15040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:47.036468 15040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:47.038887 15040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:47.039033 15040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:47.042819 15040 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:33915
I20260926 05:55:47.042860 15107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:33915 every 8 connection(s)
I20260926 05:55:47.043196 15040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:47.043459 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 15040
I20260926 05:55:47.043566 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:47.043737 25596 external_mini_cluster.cc:1428] Setting key 7b6ac9bb83d2d127add40481df2ead41
I20260926 05:55:47.044255 15108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:47.047336 15108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28: Bootstrap starting.
I20260926 05:55:47.048066 15108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:47.048338 15108 log.cc:824] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:47.049126 15108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28: No bootstrap required, opened a new log
Sep 26 05:55:47 dist-test-slave-7fr8 krb5kdc[15027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:47.051348 15108 raft_consensus.cc:348] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93e1713301846d69c9d2e2aad7d3f28" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33915 } }
I20260926 05:55:47.051506 15108 raft_consensus.cc:374] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:47.051539 15108 raft_consensus.cc:729] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93e1713301846d69c9d2e2aad7d3f28, State: Initialized, Role: FOLLOWER
I20260926 05:55:47.051667 15108 consensus_queue.cc:260] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93e1713301846d69c9d2e2aad7d3f28" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33915 } }
I20260926 05:55:47.051750 15108 raft_consensus.cc:388] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:47.051789 15108 raft_consensus.cc:482] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:47.051836 15108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:47.052397 15108 raft_consensus.cc:504] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93e1713301846d69c9d2e2aad7d3f28" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33915 } }
I20260926 05:55:47.052487 15108 leader_election.cc:302] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a93e1713301846d69c9d2e2aad7d3f28; no voters:
I20260926 05:55:47.052672 15108 leader_election.cc:288] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:47.052718 15113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:47.052883 15113 raft_consensus.cc:686] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [term 1 LEADER]: Becoming Leader. State: Replica: a93e1713301846d69c9d2e2aad7d3f28, State: Running, Role: LEADER
I20260926 05:55:47.052950 15108 sys_catalog.cc:564] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:47.052969 15113 consensus_queue.cc:237] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93e1713301846d69c9d2e2aad7d3f28" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33915 } }
I20260926 05:55:47.053390 15114 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a93e1713301846d69c9d2e2aad7d3f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93e1713301846d69c9d2e2aad7d3f28" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33915 } } }
I20260926 05:55:47.053629 15114 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:47.053395 15115 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a93e1713301846d69c9d2e2aad7d3f28. Latest consensus state: current_term: 1 leader_uuid: "a93e1713301846d69c9d2e2aad7d3f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a93e1713301846d69c9d2e2aad7d3f28" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33915 } } }
I20260926 05:55:47.053970 15115 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:47.054867 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:47.044254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37481 (local address 127.24.255.62:33915)
0926 05:55:47.044410 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:47.044419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:47.044447 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:55:47.044885 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:47.044892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:47.044937 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:47.045052 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:47.045877 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:47.047007 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:47.048009 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:47.048493 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:47.048619 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:47.051105 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:47.051109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:47.051109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:47.051122 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:47.052942 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:47.053375 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:47.053378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:47.053379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:47.053600 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:47.053847 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:47.053848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:47.053848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:47.054225 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:47.054391 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:47.054516 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 05:55:47.054668 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260926 05:55:47.054507 15122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:47.055341 15122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:47.055742 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:47.057003 15122 catalog_manager.cc:1347] Generated new cluster ID: a56840e9736347819b2c3912c7061536
I20260926 05:55:47.057058 15122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:47.061196 15122 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:47.061705 15122 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:47.064678 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:47.057166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36551 (local address 127.24.255.62:33915)
0926 05:55:47.057206 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:47.057208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:47.057215 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:55:47.057236 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:47.057237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:47.057247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:47.057303 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:47.057969 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:47.058715 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:47.059733 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:47.060029 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:47.060098 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:47.061827 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:47.061829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:47.061830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:47.061837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:47.063394 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:47.063807 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:47.063808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:47.063809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:47.063867 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:47.064064 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:47.064064 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:47.064065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:47.064189 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:47.064261 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:47.064468 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 05:55:47.064570 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:55:47.066890 15122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a93e1713301846d69c9d2e2aad7d3f28: Generated new TSK 0
I20260926 05:55:47.067067 15122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:47.069669 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 15040
2026-09-26T05:55:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:47 dist-test-slave-7fr8 krb5kdc[15139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:47 dist-test-slave-7fr8 krb5kdc[15139](info): set up 2 sockets
Sep 26 05:55:47 dist-test-slave-7fr8 krb5kdc[15139](info): commencing operation
krb5kdc: starting...
W20260926 05:55:48.042392 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:48 dist-test-slave-7fr8 krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:55:48.161839 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:39193
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:39193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:48.242429 15152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:48.242686 15152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:48.242712 15152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:48.243930 15152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:48.243974 15152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:48.243988 15152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:48.243999 15152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:48.244009 15152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:48.245764 15152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:39193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:39193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:48.246027 15152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:48.246307 15152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:48.251191 15158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:48.251317 15152 server_base.cc:1034] running on GCE node
W20260926 05:55:48.251156 15160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:48.251168 15157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:48.251689 15152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:48.252003 15152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:48.253165 15152 hybrid_clock.cc:648] HybridClock initialized: now 1790402148253130 us; error 49 us; skew 500 ppm
Sep 26 05:55:48 dist-test-slave-7fr8 krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402148, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:48.255786 15152 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:55:48.256335 15152 webserver.cc:460] Webserver started at http://127.24.255.62:41951/ using document root <none> and password file <none>
I20260926 05:55:48.256546 15152 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:48.256604 15152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:48.256739 15152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:48.257989 15152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "264510fd408a4f4790e7ea960d3c16b8"
format_stamp: "Formatted at 2026-09-26 05:55:48 on dist-test-slave-7fr8"
server_key: "5b35f96dbbc7604e2ef39be1e9f71abd"
server_key_iv: "b3d5b57932f0d0284ee2b13384b17c30"
server_key_version: "encryptionkey@0"
I20260926 05:55:48.258422 15152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "264510fd408a4f4790e7ea960d3c16b8"
format_stamp: "Formatted at 2026-09-26 05:55:48 on dist-test-slave-7fr8"
server_key: "5b35f96dbbc7604e2ef39be1e9f71abd"
server_key_iv: "b3d5b57932f0d0284ee2b13384b17c30"
server_key_version: "encryptionkey@0"
I20260926 05:55:48.259809 15152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:48.260663 15167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:48.260856 15152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:48.260947 15152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "264510fd408a4f4790e7ea960d3c16b8"
format_stamp: "Formatted at 2026-09-26 05:55:48 on dist-test-slave-7fr8"
server_key: "5b35f96dbbc7604e2ef39be1e9f71abd"
server_key_iv: "b3d5b57932f0d0284ee2b13384b17c30"
server_key_version: "encryptionkey@0"
I20260926 05:55:48.261019 15152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:48.279790 15152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:48.281941 15152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:48.282084 15152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:48.286202 15152 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:39193
I20260926 05:55:48.286240 15219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:39193 every 8 connection(s)
I20260926 05:55:48.286643 15152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:48.286782 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 15152
I20260926 05:55:48.286888 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:48.287195 25596 external_mini_cluster.cc:1428] Setting key 711fd34791ed4a6404d9b1cbc3dd3097
I20260926 05:55:48.287808 15220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:48.291108 15220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8: Bootstrap starting.
I20260926 05:55:48.291723 15220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:48.291999 15220 log.cc:824] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:48.292735 15220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8: No bootstrap required, opened a new log
I20260926 05:55:48.293231 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:48.287804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44413 (local address 127.24.255.62:39193)
0926 05:55:48.288007 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:48.288017 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:48.288048 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:55:48.288577 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:48.288587 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:48.288654 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:48.288818 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:48.289032 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:48.290423 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:48.291278 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:48.291709 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:48.291812 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:48.292583 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:48.292588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:48.292588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:48.292762 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:48.292803 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:48.292820 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:48.292879 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:48.292982 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 05:55:48.293055 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260926 05:55:48.294303 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:48.294853 15220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264510fd408a4f4790e7ea960d3c16b8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 39193 } }
I20260926 05:55:48.295002 15220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:48.295025 15220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 264510fd408a4f4790e7ea960d3c16b8, State: Initialized, Role: FOLLOWER
I20260926 05:55:48.295117 15220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264510fd408a4f4790e7ea960d3c16b8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 39193 } }
I20260926 05:55:48.295189 15220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:48.295210 15220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:48.295249 15220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:48.295852 15220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264510fd408a4f4790e7ea960d3c16b8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 39193 } }
I20260926 05:55:48.295953 15220 leader_election.cc:302] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 264510fd408a4f4790e7ea960d3c16b8; no voters:
I20260926 05:55:48.296113 15220 leader_election.cc:288] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:48.296151 15230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:48.296270 15230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [term 1 LEADER]: Becoming Leader. State: Replica: 264510fd408a4f4790e7ea960d3c16b8, State: Running, Role: LEADER
I20260926 05:55:48.296345 15220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:48.296348 15230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264510fd408a4f4790e7ea960d3c16b8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 39193 } }
I20260926 05:55:48.296725 15231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "264510fd408a4f4790e7ea960d3c16b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264510fd408a4f4790e7ea960d3c16b8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 39193 } } }
I20260926 05:55:48.296731 15232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 264510fd408a4f4790e7ea960d3c16b8. Latest consensus state: current_term: 1 leader_uuid: "264510fd408a4f4790e7ea960d3c16b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "264510fd408a4f4790e7ea960d3c16b8" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 39193 } } }
I20260926 05:55:48.296813 15231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:48.296823 15232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:48.297477 15238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:48.297973 15238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:48.299041 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:48.295675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34453 (local address 127.24.255.62:39193)
0926 05:55:48.295712 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:48.295714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:48.295823 (+ 109us) server_negotiation.cc:407] Connection header received
0926 05:55:48.295848 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:48.295849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:48.295859 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:48.295924 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:48.296022 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:48.296742 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:48.297762 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:48.298130 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:48.298203 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:48.298814 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:48.298816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:48.298816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:48.298821 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:48.298836 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:48.298840 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:48.298882 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:48.298942 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 05:55:48.298969 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:55:48.299795 15238 catalog_manager.cc:1347] Generated new cluster ID: 4b7c315ebc8c4248ae18fefe1e67a99f
I20260926 05:55:48.299850 15238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:48.318670 15238 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:48.319453 15238 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:48.323843 15238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 264510fd408a4f4790e7ea960d3c16b8: Generated new TSK 0
I20260926 05:55:48.324050 15238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:48.329972 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 15152
2026-09-26T05:55:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1260 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:48 dist-test-slave-7fr8 krb5kdc[15251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:48 dist-test-slave-7fr8 krb5kdc[15251](info): set up 2 sockets
Sep 26 05:55:48 dist-test-slave-7fr8 krb5kdc[15251](info): commencing operation
krb5kdc: starting...
W20260926 05:55:49.298439 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:49 dist-test-slave-7fr8 krb5kdc[15251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:55:49.418355 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:33759
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:33759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:49.499065 15264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:49.499305 15264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:49.499337 15264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:49.500581 15264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:49.500638 15264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:49.500655 15264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:49.500671 15264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:49.500686 15264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:49.502305 15264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:33759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:33759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:49.502594 15264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:49.502820 15264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:49.507845 15269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:49.507846 15272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:49.507865 15264 server_base.cc:1034] running on GCE node
W20260926 05:55:49.507850 15270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:49.508277 15264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:49.508577 15264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:49.509752 15264 hybrid_clock.cc:648] HybridClock initialized: now 1790402149509722 us; error 48 us; skew 500 ppm
Sep 26 05:55:49 dist-test-slave-7fr8 krb5kdc[15251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402149, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:49.517395 15264 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:55:49.517892 15264 webserver.cc:460] Webserver started at http://127.24.255.62:40003/ using document root <none> and password file <none>
I20260926 05:55:49.518144 15264 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:49.518194 15264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:49.518322 15264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:49.519448 15264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "03e29cc853384ddb934777488e937bf0"
format_stamp: "Formatted at 2026-09-26 05:55:49 on dist-test-slave-7fr8"
server_key: "5eb1b55359a77b6674785d77ef9c59e3"
server_key_iv: "d82cc2b482e762ea0cdeb2a7b223e824"
server_key_version: "encryptionkey@0"
I20260926 05:55:49.519786 15264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "03e29cc853384ddb934777488e937bf0"
format_stamp: "Formatted at 2026-09-26 05:55:49 on dist-test-slave-7fr8"
server_key: "5eb1b55359a77b6674785d77ef9c59e3"
server_key_iv: "d82cc2b482e762ea0cdeb2a7b223e824"
server_key_version: "encryptionkey@0"
I20260926 05:55:49.521190 15264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:55:49.521917 15279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:49.522061 15264 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:55:49.522212 15264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "03e29cc853384ddb934777488e937bf0"
format_stamp: "Formatted at 2026-09-26 05:55:49 on dist-test-slave-7fr8"
server_key: "5eb1b55359a77b6674785d77ef9c59e3"
server_key_iv: "d82cc2b482e762ea0cdeb2a7b223e824"
server_key_version: "encryptionkey@0"
I20260926 05:55:49.522279 15264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:49.525524 15264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:49.527549 15264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:49.527683 15264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:49.531440 15264 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:33759
I20260926 05:55:49.531479 15331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:33759 every 8 connection(s)
I20260926 05:55:49.531783 15264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:49.532723 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:49.533270 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 15264
I20260926 05:55:49.533375 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:49.533551 25596 external_mini_cluster.cc:1428] Setting key 749b9f79738d514c5e52775dc5b673c9
I20260926 05:55:49.535549 15332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0: Bootstrap starting.
I20260926 05:55:49.535954 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:49.534083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43885 (local address 127.24.255.62:33759)
0926 05:55:49.534281 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:49.534289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:49.534314 (+ 25us) server_negotiation.cc:407] Connection header received
0926 05:55:49.534719 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:49.534725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:49.534750 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:49.534865 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:49.535501 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:49.535505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:49.535505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:49.535509 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:49.535546 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:49.535557 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:49.535629 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:49.535707 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 05:55:49.535766 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260926 05:55:49.536280 15332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:49.536546 15332 log.cc:824] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:49.536954 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:49.537276 15332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0: No bootstrap required, opened a new log
I20260926 05:55:49.539523 15332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e29cc853384ddb934777488e937bf0" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33759 } }
I20260926 05:55:49.539678 15332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:55:49.539701 15332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e29cc853384ddb934777488e937bf0, State: Initialized, Role: FOLLOWER
I20260926 05:55:49.539798 15332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e29cc853384ddb934777488e937bf0" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33759 } }
I20260926 05:55:49.539887 15332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:55:49.539911 15332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:55:49.539939 15332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:49.540483 15332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e29cc853384ddb934777488e937bf0" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33759 } }
I20260926 05:55:49.540576 15332 leader_election.cc:302] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03e29cc853384ddb934777488e937bf0; no voters:
I20260926 05:55:49.540712 15332 leader_election.cc:288] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:55:49.540768 15342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:49.540920 15342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [term 1 LEADER]: Becoming Leader. State: Replica: 03e29cc853384ddb934777488e937bf0, State: Running, Role: LEADER
I20260926 05:55:49.540933 15332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:49.541208 15342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e29cc853384ddb934777488e937bf0" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33759 } }
I20260926 05:55:49.541611 15344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03e29cc853384ddb934777488e937bf0. Latest consensus state: current_term: 1 leader_uuid: "03e29cc853384ddb934777488e937bf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e29cc853384ddb934777488e937bf0" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33759 } } }
I20260926 05:55:49.541777 15344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:49.541602 15343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03e29cc853384ddb934777488e937bf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e29cc853384ddb934777488e937bf0" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33759 } } }
I20260926 05:55:49.541955 15343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:49.542037 15346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:49.542263 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:49.538551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57345 (local address 127.24.255.62:33759)
0926 05:55:49.540994 (+ 2443us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:49.540996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:49.541007 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:55:49.541031 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:49.541032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:49.541036 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:49.541103 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:49.541935 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:49.541938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:49.541938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:55:49.541943 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:49.541961 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:55:49.541966 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:49.542026 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:49.542179 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 05:55:49.542197 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2421}
I20260926 05:55:49.542596 15346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:49.544313 15346 catalog_manager.cc:1347] Generated new cluster ID: 87b7957d08924262aea3797f2a915392
I20260926 05:55:49.544365 15346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:49.555258 15346 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:49.555428 15346 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:49.558166 15346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03e29cc853384ddb934777488e937bf0: Generated new TSK 0
I20260926 05:55:49.558339 15346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:49.561890 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 15264
2026-09-26T05:55:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1232 ms)
[----------] 18 tests from AuthTokenIssuingTest (22941 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:49 dist-test-slave-7fr8 krb5kdc[15363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:49 dist-test-slave-7fr8 krb5kdc[15363](info): set up 2 sockets
Sep 26 05:55:49 dist-test-slave-7fr8 krb5kdc[15363](info): commencing operation
krb5kdc: starting...
W20260926 05:55:50.534397 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:50 dist-test-slave-7fr8 krb5kdc[15363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:55:50.663800 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:42395
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:42395,127.24.255.61:44759,127.24.255.60:36759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:50.749733 15376 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:55:50.749965 15376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:50.749982 15376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:50.750002 15376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:50.751253 15376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:50.751302 15376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:50.751314 15376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:50.751325 15376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:50.751335 15376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:50.752946 15376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:42395,127.24.255.61:44759,127.24.255.60:36759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:42395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:50.753221 15376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:50.753458 15376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:50.758400 15384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:50.758400 15381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:50.758394 15382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:50.758662 15376 server_base.cc:1034] running on GCE node
I20260926 05:55:50.758889 15376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:50.759150 15376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:50.760304 15376 hybrid_clock.cc:648] HybridClock initialized: now 1790402150760283 us; error 36 us; skew 500 ppm
Sep 26 05:55:50 dist-test-slave-7fr8 krb5kdc[15363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402150, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:50.763965 15376 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:55:50.764504 15376 webserver.cc:460] Webserver started at http://127.24.255.62:35605/ using document root <none> and password file <none>
I20260926 05:55:50.764753 15376 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:50.764804 15376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:50.764941 15376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:50.766031 15376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "f6b6628f132e47d3ae7e0ca01f63aab9"
format_stamp: "Formatted at 2026-09-26 05:55:50 on dist-test-slave-7fr8"
server_key: "b1e59b811111a663cb3807619bddcc97"
server_key_iv: "07a85d130d43e630673bb4e243198ed5"
server_key_version: "encryptionkey@0"
I20260926 05:55:50.766414 15376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "f6b6628f132e47d3ae7e0ca01f63aab9"
format_stamp: "Formatted at 2026-09-26 05:55:50 on dist-test-slave-7fr8"
server_key: "b1e59b811111a663cb3807619bddcc97"
server_key_iv: "07a85d130d43e630673bb4e243198ed5"
server_key_version: "encryptionkey@0"
I20260926 05:55:50.767764 15376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:50.768532 15391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:50.768674 15376 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:55:50.768761 15376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "f6b6628f132e47d3ae7e0ca01f63aab9"
format_stamp: "Formatted at 2026-09-26 05:55:50 on dist-test-slave-7fr8"
server_key: "b1e59b811111a663cb3807619bddcc97"
server_key_iv: "07a85d130d43e630673bb4e243198ed5"
server_key_version: "encryptionkey@0"
I20260926 05:55:50.768836 15376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:50.792305 15376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:50.795203 15376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:50.795356 15376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:50.799268 15376 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:42395
I20260926 05:55:50.799304 15443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:42395 every 8 connection(s)
I20260926 05:55:50.799670 15376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:50.800642 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:50.802245 15444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:55:50 dist-test-slave-7fr8 krb5kdc[15363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402150, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:50.809566 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 15376
I20260926 05:55:50.809662 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:50.809837 25596 external_mini_cluster.cc:1428] Setting key 9bcfb1ab3b3b8c49e1122d4bb1f7e6bd
I20260926 05:55:50.813024 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:50.803038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.24.255.62:42395)
0926 05:55:50.803194 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:50.803202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:50.803306 (+ 104us) server_negotiation.cc:407] Connection header received
0926 05:55:50.803858 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:50.803860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:50.803877 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:50.803973 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:50.804630 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:50.805894 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:50.806991 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:50.807586 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:50.807690 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:50.809926 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:50.809936 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:50.809936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:50.809944 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:50.811524 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:50.811889 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:50.811892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:50.811893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:50.811968 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:50.812160 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:50.812161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:50.812161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:50.812392 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:50.812462 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:50.812648 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 05:55:50.812760 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260926 05:55:50.813026 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:50.802989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:42395 (local address 127.0.0.1:34640)
0926 05:55:50.803187 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:50.803227 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:50.803708 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:50.803984 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:50.803987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:50.804250 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:50.804414 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:50.804418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:50.806015 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:50.806016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:50.806854 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:50.806857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:50.807714 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:50.807715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:50.807777 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:50.808295 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:50.808303 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:50.809838 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:50.811621 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:50.811627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:50.811629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:50.811816 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:50.812025 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:50.812026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:50.812026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:50.812110 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:50.812452 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:50.812457 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:50.812522 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 05:55:50.812605 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 05:55:50.812834 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":80,"thread_start_us":102,"threads_started":1}
I20260926 05:55:50.814028 15444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:55:50.814414 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:50.814306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:44759 (local address 127.0.0.1:57788)
0926 05:55:50.814351 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:50.814392 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:55:50.814600 15394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.255.61:44759: connect: Connection refused (error 111)
I20260926 05:55:50.815011 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:50.814918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:44759 (local address 127.0.0.1:57802)
0926 05:55:50.814958 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:50.814989 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 05:55:50.815105 15444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.61:44759: Network error: Client connection negotiation failed: client connection to 127.24.255.61:44759: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.24.255.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.61.keytab.
Entry for principal kudu/127.24.255.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.61.keytab.
I20260926 05:55:50.838212 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.61:44759
--webserver_interface=127.24.255.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.61.keytab
--principal=kudu/127.24.255.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:42395,127.24.255.61:44759,127.24.255.60:36759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:55:50.864362 15444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } attempt: 1
I20260926 05:55:50.864982 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:50.864773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:44759 (local address 127.0.0.1:57816)
0926 05:55:50.864840 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:50.864943 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 05:55:50.865386 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:50.865286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:44759 (local address 127.0.0.1:57822)
0926 05:55:50.865332 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:50.865364 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 05:55:50.865487 15444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.61:44759: Network error: Client connection negotiation failed: client connection to 127.24.255.61:44759: connect: Connection refused (error 111)
W20260926 05:55:50.924029 15451 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:55:50.924273 15451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:50.924290 15451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:50.924312 15451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:50.925611 15451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:50.925671 15451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:50.925684 15451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:50.925696 15451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:50.925709 15451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:50.927356 15451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:42395,127.24.255.61:44759,127.24.255.60:36759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.61.keytab
--principal=kudu/127.24.255.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.61:44759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.255.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:50.927603 15451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:50.927831 15451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:50.932965 15456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:50.932965 15457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:50.933049 15459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:50.933295 15451 server_base.cc:1034] running on GCE node
I20260926 05:55:50.933535 15451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:50.933660 15444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } attempt: 2
I20260926 05:55:50.933879 15451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:50.934078 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:50.933929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:44759 (local address 127.0.0.1:57828)
0926 05:55:50.933993 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:50.934022 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 05:55:50.934386 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:50.934313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:44759 (local address 127.0.0.1:57842)
0926 05:55:50.934339 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:50.934366 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 05:55:50.934460 15444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.61:44759: Network error: Client connection negotiation failed: client connection to 127.24.255.61:44759: connect: Connection refused (error 111)
I20260926 05:55:50.935030 15451 hybrid_clock.cc:648] HybridClock initialized: now 1790402150935009 us; error 28 us; skew 500 ppm
Sep 26 05:55:50 dist-test-slave-7fr8 krb5kdc[15363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402150, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:50.937549 15451 init.cc:377] Logged in from keytab as kudu/127.24.255.61@KRBTEST.COM (short username kudu)
I20260926 05:55:50.938031 15451 webserver.cc:460] Webserver started at http://127.24.255.61:42805/ using document root <none> and password file <none>
I20260926 05:55:50.938310 15451 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:50.938360 15451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:50.938481 15451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:50.939560 15451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/data/instance:
uuid: "d00606e9f96443eda5038abca15cd49d"
format_stamp: "Formatted at 2026-09-26 05:55:50 on dist-test-slave-7fr8"
server_key: "2736362da9c9c7cb9bde47ab6c5144f7"
server_key_iv: "19cbf600345c9168f586e67d7f5683c3"
server_key_version: "encryptionkey@0"
I20260926 05:55:50.939890 15451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/wal/instance:
uuid: "d00606e9f96443eda5038abca15cd49d"
format_stamp: "Formatted at 2026-09-26 05:55:50 on dist-test-slave-7fr8"
server_key: "2736362da9c9c7cb9bde47ab6c5144f7"
server_key_iv: "19cbf600345c9168f586e67d7f5683c3"
server_key_version: "encryptionkey@0"
I20260926 05:55:50.941228 15451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:55:50.942005 15466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:50.942233 15451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:50.942324 15451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/wal
uuid: "d00606e9f96443eda5038abca15cd49d"
format_stamp: "Formatted at 2026-09-26 05:55:50 on dist-test-slave-7fr8"
server_key: "2736362da9c9c7cb9bde47ab6c5144f7"
server_key_iv: "19cbf600345c9168f586e67d7f5683c3"
server_key_version: "encryptionkey@0"
I20260926 05:55:50.942384 15451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:50.968873 15451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:50.971165 15451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:50.971289 15451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:50.975272 15451 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.61:44759
I20260926 05:55:50.975306 15518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.61:44759 every 8 connection(s)
I20260926 05:55:50.975654 15451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/data/info.pb
I20260926 05:55:50.976657 15519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:50.978283 15519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:55:50.983779 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 15451
I20260926 05:55:50.983892 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-1/wal/instance
I20260926 05:55:50.984077 25596 external_mini_cluster.cc:1428] Setting key 0d1c1c0783e3ede1b1f46d81467b6edd
Sep 26 05:55:50 dist-test-slave-7fr8 krb5kdc[15363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402150, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.61@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:50.988268 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:50.979308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.24.255.62:42395)
0926 05:55:50.979424 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:50.979427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:50.979795 (+ 368us) server_negotiation.cc:407] Connection header received
0926 05:55:50.980321 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:50.980323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:50.980340 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:50.980394 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:50.981059 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:50.981554 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:50.982646 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:50.982858 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:50.982914 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:50.985410 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:50.985413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:50.985413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:50.985420 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:50.986822 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:50.987227 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:50.987229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:50.987229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:50.987294 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:50.987572 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:50.987573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:50.987573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:50.987718 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:50.987779 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:50.988053 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 05:55:50.988160 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 05:55:50.988412 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:50.979228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:42395 (local address 127.0.0.1:34646)
0926 05:55:50.979621 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:50.979673 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:50.980217 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:50.980469 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:50.980472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:50.980750 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:50.980961 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:50.980965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:50.981641 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:50.981643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:50.982561 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:50.982564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:50.982939 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:50.982941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:50.983080 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:50.983692 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:50.983702 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:50.985284 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:50.986908 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:50.986916 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:50.986919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:50.987147 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:50.987399 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:50.987400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:50.987400 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:50.987513 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:50.987776 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:50.987781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:50.987930 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 05:55:50.988018 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 05:55:50.988190 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":188,"threads_started":1}
I20260926 05:55:50.988982 15519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.255.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.60@KRBTEST.COM" created.
Sep 26 05:55:50 dist-test-slave-7fr8 krb5kdc[15363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402150, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.61@KRBTEST.COM for kudu/127.24.255.61@KRBTEST.COM
I20260926 05:55:50.999261 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:50.989289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57856 (local address 127.24.255.61:44759)
0926 05:55:50.989433 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:50.989437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:50.989454 (+ 17us) server_negotiation.cc:407] Connection header received
0926 05:55:50.989472 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:50.989474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:50.989485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:50.989541 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:50.989848 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:50.990596 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:50.991574 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:50.991914 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:50.991977 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:50.996128 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:50.996143 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:50.996144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:50.996156 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:50.997928 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:50.998418 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:50.998424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:50.998425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:50.998487 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:50.998686 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:50.998686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:50.998686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:50.998863 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:50.998935 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:50.999089 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 05:55:50.999176 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260926 05:55:50.999374 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:50.989238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:44759 (local address 127.0.0.1:57856)
0926 05:55:50.989291 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:50.989303 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:50.989350 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:50.989560 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:50.989561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:50.989711 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:50.989780 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:50.989781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:50.990680 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:50.990681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:50.991479 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:50.991482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:50.991997 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:50.991998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:50.992077 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:50.992519 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:50.992524 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:50.995979 (+ 3455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:50.998027 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:50.998029 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:50.998029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:50.998285 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:50.998564 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:50.998565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:50.998565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:50.998630 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:50.998933 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:50.998934 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:50.998986 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 05:55:50.999058 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:55:50.999254 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":76}
I20260926 05:55:51.000156 15519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:55:51.000514 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.000380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:36759 (local address 127.0.0.1:52800)
0926 05:55:51.000429 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.000484 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 05:55:51.000725 15467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
I20260926 05:55:51.001132 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.001029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:36759 (local address 127.0.0.1:52812)
0926 05:55:51.001076 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.001108 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:55:51.001235 15519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:36759: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.60.keytab.
Entry for principal kudu/127.24.255.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.60.keytab.
I20260926 05:55:51.014348 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.60:36759
--webserver_interface=127.24.255.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.60.keytab
--principal=kudu/127.24.255.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:42395,127.24.255.61:44759,127.24.255.60:36759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:55:51.025658 15444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } attempt: 3
Sep 26 05:55:51 dist-test-slave-7fr8 krb5kdc[15363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402150, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for kudu/127.24.255.61@KRBTEST.COM
I20260926 05:55:51.033999 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.026388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.24.255.61:44759)
0926 05:55:51.026512 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.026514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.026530 (+ 16us) server_negotiation.cc:407] Connection header received
0926 05:55:51.026567 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.026569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.026587 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.026677 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.027159 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.027719 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.028712 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.028995 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.029074 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.031435 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:51.031437 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:51.031438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:51.031446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:51.032893 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.033254 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.033256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.033256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.033309 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.033473 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.033474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.033474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.033606 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:51.033662 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:51.033805 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 05:55:51.033897 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 05:55:51.033999 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.026088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:44759 (local address 127.0.0.1:57862)
0926 05:55:51.026238 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.026253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:51.026324 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:51.026727 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:51.026729 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:51.026970 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:51.027065 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:51.027068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.027802 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.027803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:51.028532 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:51.028539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.029082 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.029084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:51.029244 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.029763 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:51.029768 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:51.031314 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:51.033011 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:51.033013 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:51.033013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:51.033188 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:51.033366 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:51.033367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:51.033367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:51.033422 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:51.033663 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:51.033665 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:51.033719 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 05:55:51.033778 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 05:55:51.033894 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260926 05:55:51.034899 15444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:55:51.035334 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.035220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:36759 (local address 127.0.0.1:52826)
0926 05:55:51.035272 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.035303 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 05:55:51.035650 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.035569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:36759 (local address 127.0.0.1:52840)
0926 05:55:51.035603 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.035627 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 05:55:51.035732 15444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:36759: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
I20260926 05:55:51.050477 15519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } attempt: 1
I20260926 05:55:51.051055 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.050891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:36759 (local address 127.0.0.1:52846)
0926 05:55:51.050959 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.051012 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:55:51.051462 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.051376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:36759 (local address 127.0.0.1:52854)
0926 05:55:51.051414 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.051443 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 05:55:51.051641 15519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:36759: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
I20260926 05:55:51.066931 15444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } attempt: 1
I20260926 05:55:51.067432 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.067305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:36759 (local address 127.0.0.1:52856)
0926 05:55:51.067369 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.067401 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 05:55:51.067786 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.067705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:36759 (local address 127.0.0.1:52864)
0926 05:55:51.067741 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.067767 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 05:55:51.067875 15444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:36759: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
W20260926 05:55:51.101779 15526 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:55:51.102039 15526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:51.102065 15526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:51.102120 15526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:51.103336 15526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:51.103391 15526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:51.103408 15526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:51.103425 15526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:51.103440 15526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:51.105031 15526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:42395,127.24.255.61:44759,127.24.255.60:36759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.60.keytab
--principal=kudu/127.24.255.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.60:36759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.255.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:51.105314 15526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:51.105537 15526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:51.111001 15534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:51.111086 15526 server_base.cc:1034] running on GCE node
W20260926 05:55:51.111020 15532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:51.111025 15531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:51.111505 15526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:51.111811 15526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:51.113056 15526 hybrid_clock.cc:648] HybridClock initialized: now 1790402151113027 us; error 34 us; skew 500 ppm
Sep 26 05:55:51 dist-test-slave-7fr8 krb5kdc[15363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402151, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:51.116344 15526 init.cc:377] Logged in from keytab as kudu/127.24.255.60@KRBTEST.COM (short username kudu)
I20260926 05:55:51.116839 15526 webserver.cc:460] Webserver started at http://127.24.255.60:45493/ using document root <none> and password file <none>
I20260926 05:55:51.117049 15526 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:51.117098 15526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:51.117235 15526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:51.118414 15526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/data/instance:
uuid: "53342f81df484088a5bf786ca3694cb2"
format_stamp: "Formatted at 2026-09-26 05:55:51 on dist-test-slave-7fr8"
server_key: "50eb581f0fe66a906d0e6d3055108f50"
server_key_iv: "f1e6f609845e9fea66bfd68b85057f24"
server_key_version: "encryptionkey@0"
I20260926 05:55:51.118760 15526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/wal/instance:
uuid: "53342f81df484088a5bf786ca3694cb2"
format_stamp: "Formatted at 2026-09-26 05:55:51 on dist-test-slave-7fr8"
server_key: "50eb581f0fe66a906d0e6d3055108f50"
server_key_iv: "f1e6f609845e9fea66bfd68b85057f24"
server_key_version: "encryptionkey@0"
I20260926 05:55:51.119845 15519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } attempt: 2
I20260926 05:55:51.120230 15526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:51.120321 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.120194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:36759 (local address 127.0.0.1:52866)
0926 05:55:51.120252 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.120286 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 05:55:51.120687 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.120597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:36759 (local address 127.0.0.1:52882)
0926 05:55:51.120634 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.120660 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 05:55:51.120776 15519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:36759: Network error: Client connection negotiation failed: client connection to 127.24.255.60:36759: connect: Connection refused (error 111)
I20260926 05:55:51.121052 15541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:51.121239 15526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:51.121330 15526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/wal
uuid: "53342f81df484088a5bf786ca3694cb2"
format_stamp: "Formatted at 2026-09-26 05:55:51 on dist-test-slave-7fr8"
server_key: "50eb581f0fe66a906d0e6d3055108f50"
server_key_iv: "f1e6f609845e9fea66bfd68b85057f24"
server_key_version: "encryptionkey@0"
I20260926 05:55:51.121397 15526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:51.133100 15526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:51.135306 15526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:51.135456 15526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:51.139185 15526 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.60:36759
I20260926 05:55:51.139222 15593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.60:36759 every 8 connection(s)
I20260926 05:55:51.139550 15526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/data/info.pb
I20260926 05:55:51.140116 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 15526
I20260926 05:55:51.140250 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401881171543-25596-0/minicluster-data/master-2/wal/instance
I20260926 05:55:51.140451 25596 external_mini_cluster.cc:1428] Setting key 7ac1723525cc40ba4724471a7f3aa57a
I20260926 05:55:51.140574 15594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:51.142755 15594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:55:51.143038 15444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } attempt: 2
Sep 26 05:55:51 dist-test-slave-7fr8 krb5kdc[15363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402150, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for kudu/127.24.255.60@KRBTEST.COM
Sep 26 05:55:51 dist-test-slave-7fr8 krb5kdc[15363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402151, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.60@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
Sep 26 05:55:51 dist-test-slave-7fr8 krb5kdc[15363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.60@KRBTEST.COM
I20260926 05:55:51.151690 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.143306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:36759 (local address 127.0.0.1:52900)
0926 05:55:51.143425 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.143437 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:51.143480 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:51.143731 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:51.143732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:51.143894 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:51.143954 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:51.143956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.144612 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.144613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:51.145365 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:51.145367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.146026 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.146027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:51.146078 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.146602 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:51.146606 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:51.147995 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:51.150458 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:51.150460 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:51.150460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:51.150627 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:51.150854 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:51.150855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:51.150855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:51.150911 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:51.151286 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:51.151287 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:51.151335 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 05:55:51.151482 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 05:55:51.151584 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 05:55:51.151925 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.143345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52900 (local address 127.24.255.60:36759)
0926 05:55:51.143628 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.143630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.143637 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:55:51.143655 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.143656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.143664 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.143720 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.144005 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.144540 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.145431 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.145795 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.145900 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.148081 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:51.148089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:51.148089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:51.148097 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:51.150384 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.150697 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.150701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.150701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.150793 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.150967 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.150967 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.150968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.151224 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:51.151295 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:51.151565 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 05:55:51.151694 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":160,"threads_started":1}
I20260926 05:55:51.152194 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.143572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34660 (local address 127.24.255.62:42395)
0926 05:55:51.143623 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.143625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.143761 (+ 136us) server_negotiation.cc:407] Connection header received
0926 05:55:51.143854 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.143854 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.143863 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.143913 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.144496 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.145198 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.146139 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.146375 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.146429 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.149207 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:51.149208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:51.149208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:51.149216 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:51.150922 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.151381 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.151382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.151383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.151451 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.151646 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.151647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.151647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.151756 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:51.151798 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:51.152041 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 05:55:51.152115 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:55:51.152508 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.143358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:42395 (local address 127.0.0.1:34660)
0926 05:55:51.143664 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.143699 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:51.143793 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:51.143941 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:51.143943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:51.144261 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:51.144373 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:51.144377 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.145307 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.145308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:51.146042 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:51.146045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.146478 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.146480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:51.146556 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.146923 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:51.146933 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:51.149126 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:51.151027 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:51.151035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:51.151039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:51.151315 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:51.151518 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:51.151519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:51.151520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:51.151580 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:51.151844 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:51.151850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:51.151897 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 05:55:51.152300 (+ 403us) client_negotiation.cc:240] Negotiation successful
0926 05:55:51.152416 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":85,"threads_started":1}
I20260926 05:55:51.153371 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.141224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.24.255.60:36759)
0926 05:55:51.141414 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.141424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.141458 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:55:51.142007 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.142017 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.142080 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.142261 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.142865 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.144378 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.145229 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.145795 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.145900 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.149206 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:51.149208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:51.149208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:51.149216 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:51.151641 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.152045 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.152046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.152046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.152099 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.152305 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.152305 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.152306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.152425 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:51.152566 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:51.153182 (+ 616us) server_negotiation.cc:299] Negotiation successful
0926 05:55:51.153261 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260926 05:55:51.153858 15594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:55:51.154330 15444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9: Bootstrap starting.
I20260926 05:55:51.155087 15444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:51.155380 15444 log.cc:824] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:51.156275 15444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9: No bootstrap required, opened a new log
Sep 26 05:55:51 dist-test-slave-7fr8 krb5kdc[15363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402151, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.60@KRBTEST.COM for kudu/127.24.255.61@KRBTEST.COM
I20260926 05:55:51.158778 15444 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } }
I20260926 05:55:51.158933 15444 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b6628f132e47d3ae7e0ca01f63aab9, State: Initialized, Role: FOLLOWER
I20260926 05:55:51.159024 15444 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } }
I20260926 05:55:51.159396 15444 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:51.159682 15601 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } } }
I20260926 05:55:51.159760 15601 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 05:55:51 dist-test-slave-7fr8 krb5kdc[15363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:51.161643 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.154233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.24.255.61:44759)
0926 05:55:51.154285 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.154287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.154402 (+ 115us) server_negotiation.cc:407] Connection header received
0926 05:55:51.154482 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.154483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.154494 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.154546 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.154853 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.155397 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.156253 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.156533 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.156577 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.158861 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:51.158863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:51.158863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:51.158872 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:51.160344 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.160757 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.160759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.160759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.160830 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.161053 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.161055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.161055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.161213 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:51.161277 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:51.161472 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 05:55:51.161552 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 05:55:51.161661 15612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 05:55:51.161931 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.154133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:44759 (local address 127.0.0.1:57876)
0926 05:55:51.154358 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.154367 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:51.154428 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:51.154550 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:51.154550 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:51.154707 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:51.154773 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:51.154775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.155471 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.155472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:51.156151 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:51.156155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.156655 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.156656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:51.156731 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.157173 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:51.157178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:51.158735 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:51.160429 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:51.160430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:51.160430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:51.160648 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:51.160917 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:51.160918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:51.160918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:51.160982 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:51.161306 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:51.161307 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:51.161356 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 05:55:51.161724 (+ 368us) client_negotiation.cc:240] Negotiation successful
0926 05:55:51.161834 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260926 05:55:51.162638 15594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } has no permanent_uuid. Determining permanent_uuid...
W20260926 05:55:51.161979 15612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:55:51.164088 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.154136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34670 (local address 127.24.255.62:42395)
0926 05:55:51.154170 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.154171 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.154282 (+ 111us) server_negotiation.cc:407] Connection header received
0926 05:55:51.154339 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.154340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.154349 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.154399 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.154887 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.155538 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.157359 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.157642 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.157702 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.160908 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:51.160910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:51.160910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:51.160918 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:51.162648 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.163022 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.163024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.163024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.163081 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.163283 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.163284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.163285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.163409 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:51.163543 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:51.163912 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 05:55:51.163992 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 05:55:51.162935 15376 master_runner.cc:186] Error getting master registration for 127.24.255.61:44759: OK, instance_id { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" instance_seqno: 1790402150974176 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:55:51.165477 15376 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 05:55:51 dist-test-slave-7fr8 krb5kdc[15363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402151, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.60@KRBTEST.COM for kudu/127.24.255.60@KRBTEST.COM
Sep 26 05:55:51 dist-test-slave-7fr8 krb5kdc[15363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.61@KRBTEST.COM
I20260926 05:55:51.170591 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.162951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.24.255.60:36759)
0926 05:55:51.163035 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.163037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.163044 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:55:51.163062 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.163063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.163072 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.163123 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.163654 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.164370 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.165260 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.165541 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.165585 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.167659 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:51.167661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:51.167661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:51.167669 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:51.169320 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.169702 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.169703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.169704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.169763 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.169942 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.169942 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.169943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.170185 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:51.170250 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:51.170411 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 05:55:51.170493 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 05:55:51.170922 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.162896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:36759 (local address 127.0.0.1:52906)
0926 05:55:51.162938 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.162949 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:51.162987 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:51.163132 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:51.163133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:51.163278 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:51.163323 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:51.163324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.164459 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.164460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:51.165173 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:51.165175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.165668 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.165668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:51.165721 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.166238 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:51.166244 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:51.167568 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:51.169407 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:51.169408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:51.169408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:51.169627 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:51.169826 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:51.169826 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:51.169827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:51.169881 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:51.170272 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:51.170273 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:51.170313 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 05:55:51.170748 (+ 435us) client_negotiation.cc:240] Negotiation successful
0926 05:55:51.170842 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 05:55:51.172739 15594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2: Bootstrap starting.
I20260926 05:55:51.173640 15594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:51.173956 15594 log.cc:824] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:51.174283 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.166216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57882 (local address 127.24.255.61:44759)
0926 05:55:51.166259 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.166261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.166268 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:55:51.166292 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.166293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.166302 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.166356 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.166976 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.167654 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.168428 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.168648 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.168701 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.170964 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:51.170966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:51.170966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:51.170973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:51.172559 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.172941 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.172958 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.172959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.173014 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.173227 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.173228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.173228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.173351 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:51.173594 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:51.174064 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 05:55:51.174195 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:55:51.175009 15594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2: No bootstrap required, opened a new log
I20260926 05:55:51.175770 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:51.176880 15418 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34670
I20260926 05:55:51.177050 15418 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 05:55:51.177110 15418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:51.177068 15594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } }
I20260926 05:55:51.177225 15594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53342f81df484088a5bf786ca3694cb2, State: Initialized, Role: FOLLOWER
I20260926 05:55:51.177305 15594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } }
I20260926 05:55:51.177645 15594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:51.177589 15615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } } }
I20260926 05:55:51.177691 15615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:55:51.178350 15418 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } }
I20260926 05:55:51.178790 15418 leader_election.cc:288] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [CANDIDATE]: Term 1 election: Requested vote from peers d00606e9f96443eda5038abca15cd49d (127.24.255.61:44759), 53342f81df484088a5bf786ca3694cb2 (127.24.255.60:36759)
I20260926 05:55:51.179667 15568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "53342f81df484088a5bf786ca3694cb2"
I20260926 05:55:51.179803 15568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:51.180732 15568 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6628f132e47d3ae7e0ca01f63aab9 in term 1.
I20260926 05:55:51.180917 15392 leader_election.cc:302] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53342f81df484088a5bf786ca3694cb2, f6b6628f132e47d3ae7e0ca01f63aab9; no voters:
I20260926 05:55:51.181072 15601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:51.181257 15601 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [term 1 LEADER]: Becoming Leader. State: Replica: f6b6628f132e47d3ae7e0ca01f63aab9, State: Running, Role: LEADER
I20260926 05:55:51.181352 15601 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } }
W20260926 05:55:51.181571 15626 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:55:51.181617 15626 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:55:51.182173 15631 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6b6628f132e47d3ae7e0ca01f63aab9. Latest consensus state: current_term: 1 leader_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } } }
I20260926 05:55:51.182261 15631 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:51.182279 15526 master_runner.cc:186] Error getting master registration for 127.24.255.61:44759: OK, instance_id { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" instance_seqno: 1790402150974176 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:55:51.184525 15526 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:55:51.190743 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.182181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.24.255.62:42395)
0926 05:55:51.182217 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.182219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.182561 (+ 342us) server_negotiation.cc:407] Connection header received
0926 05:55:51.182657 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.182658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.182669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.182698 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.183458 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.184175 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.185266 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.185584 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.185664 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.187395 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:51.187397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:51.187397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:51.187406 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:51.189268 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.189697 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.189698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.189699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.189762 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.189958 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.189959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.189960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.190218 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:51.190332 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:51.190534 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 05:55:51.190640 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:55:51.194293 15635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:51.194654 15635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:51.198194 15568 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2 [term 1 FOLLOWER]: Refusing update from remote peer f6b6628f132e47d3ae7e0ca01f63aab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:51.198513 15631 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:51.198964 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.181960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.24.255.61:44759)
0926 05:55:51.182003 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.182005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.191234 (+ 9229us) server_negotiation.cc:407] Connection header received
0926 05:55:51.191298 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.191299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.191312 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.191372 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.191945 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.192685 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.193684 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.193988 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.194055 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.195533 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:51.195535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:51.195536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:51.195544 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:51.197356 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.197761 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.197763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.197763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.197834 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.198053 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.198054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.198055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.198216 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:51.198301 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:51.198773 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 05:55:51.198868 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:55:51.200023 15615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6b6628f132e47d3ae7e0ca01f63aab9. Latest consensus state: current_term: 1 leader_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } } }
I20260926 05:55:51.200137 15615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:55:51.201655 15601 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } } }
I20260926 05:55:51.201751 15601 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:51.201882 15615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } } }
I20260926 05:55:51.201972 15615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:55:51.202234 15637 mvcc.cc:205] Tried to move back new op lower bound from 7333487211300937728 to 7333487211239493632. Current Snapshot: MvccSnapshot[applied={T|T < 7333487211300937728}]
I20260926 05:55:51.202375 15638 mvcc.cc:205] Tried to move back new op lower bound from 7333487211300937728 to 7333487211239493632. Current Snapshot: MvccSnapshot[applied={T|T < 7333487211300937728}]
I20260926 05:55:51.202934 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.194368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.24.255.60:36759)
0926 05:55:51.194432 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.194434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.195693 (+ 1259us) server_negotiation.cc:407] Connection header received
0926 05:55:51.195793 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.195794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.195814 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.195845 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.196455 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.197282 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.198210 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.198473 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.198513 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.200149 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:51.200150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:51.200151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:51.200158 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:51.201659 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.202033 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.202034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.202035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.202092 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.202330 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.202331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.202331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.202469 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:51.202545 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:51.202775 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 05:55:51.202853 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:55:51.203334 15635 catalog_manager.cc:1347] Generated new cluster ID: fccc273487714a4e87488339d1742781
I20260926 05:55:51.203485 15635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:51.211921 15519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } attempt: 3
Sep 26 05:55:51 dist-test-slave-7fr8 krb5kdc[15363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402150, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.61@KRBTEST.COM for kudu/127.24.255.60@KRBTEST.COM
I20260926 05:55:51.222718 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.212277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:36759 (local address 127.0.0.1:52928)
0926 05:55:51.212374 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:51.212385 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:51.212425 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:51.212684 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:51.212685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:51.212875 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:51.212947 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:51.212949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.214808 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.214809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:51.215654 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:51.215657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.216285 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.216286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:51.216364 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.216956 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:51.216961 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:51.218522 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:51.221490 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:51.221492 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:51.221492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:51.221724 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:51.221961 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:51.221962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:51.221963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:51.222032 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:51.222373 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:51.222374 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:51.222432 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 05:55:51.222491 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 05:55:51.222601 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 05:55:51.223133 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:51.212499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.24.255.60:36759)
0926 05:55:51.212539 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.212541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.212548 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:55:51.212571 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.212572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.212582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.212626 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.213024 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.213780 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.215825 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:51.216145 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:51.216206 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:51.218624 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:51.218626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:51.218626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:51.218635 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:51.221379 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.221816 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.221817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.221817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.221889 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:51.222130 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:51.222131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:51.222131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:51.222273 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:51.222333 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:51.222926 (+ 593us) server_negotiation.cc:299] Negotiation successful
0926 05:55:51.223010 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:55:51.224829 15519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d: Bootstrap starting.
I20260926 05:55:51.225649 15519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:51.225958 15519 log.cc:824] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:51.238662 15635 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:51.238895 15519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d: No bootstrap required, opened a new log
I20260926 05:55:51.240059 15635 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:51.240906 15519 raft_consensus.cc:348] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } }
I20260926 05:55:51.241058 15519 raft_consensus.cc:729] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00606e9f96443eda5038abca15cd49d, State: Initialized, Role: FOLLOWER
I20260926 05:55:51.241137 15519 consensus_queue.cc:260] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } }
I20260926 05:55:51.241492 15519 sys_catalog.cc:564] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:51.241573 15492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:51.241537 15493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d00606e9f96443eda5038abca15cd49d"
I20260926 05:55:51.243799 15492 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d [term 1 FOLLOWER]: Refusing update from remote peer f6b6628f132e47d3ae7e0ca01f63aab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:51.243947 15643 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } } }
I20260926 05:55:51.244035 15643 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:55:51.245355 15493 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6628f132e47d3ae7e0ca01f63aab9 in term 1.
I20260926 05:55:51.245947 15601 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:51.247884 15643 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6b6628f132e47d3ae7e0ca01f63aab9. Latest consensus state: current_term: 1 leader_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } } }
I20260926 05:55:51.247969 15643 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:55:51.248001 15643 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } } }
I20260926 05:55:51.248036 15643 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:55:51.250752 15654 catalog_manager.cc:1259] Loaded cluster ID: fccc273487714a4e87488339d1742781
I20260926 05:55:51.250816 15654 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d: loading cluster ID for follower catalog manager: success
I20260926 05:55:51.252068 15654 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d: acquiring CA information for follower catalog manager: success
W20260926 05:55:51.252203 15654 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260926 05:55:51.253005 15635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9: Generated new TSK 0
I20260926 05:55:51.253252 15635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:51.254899 15631 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6628f132e47d3ae7e0ca01f63aab9" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 42395 } } peers { permanent_uuid: "d00606e9f96443eda5038abca15cd49d" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 44759 } } peers { permanent_uuid: "53342f81df484088a5bf786ca3694cb2" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 36759 } } }
I20260926 05:55:51.255009 15631 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b6628f132e47d3ae7e0ca01f63aab9 [sys.catalog]: This master's current role is: LEADER
W20260926 05:55:51.283787 15447 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:51.282372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.24.255.62:42395)
0926 05:55:51.282418 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.282420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.283080 (+ 660us) server_negotiation.cc:407] Connection header received
0926 05:55:51.283206 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.283207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.283218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.283249 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.283707 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34696: BlockingRecv error: recv got EOF from 127.0.0.1:34696 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:55:51.284698 15523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:51.282781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.24.255.61:44759)
0926 05:55:51.282824 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.282826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.284183 (+ 1357us) server_negotiation.cc:407] Connection header received
0926 05:55:51.284281 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.284283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.284294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.284321 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.284635 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57894: BlockingRecv error: recv got EOF from 127.0.0.1:57894 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:55:51.286006 15597 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:51.285207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.24.255.60:36759)
0926 05:55:51.285249 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.285251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.285362 (+ 111us) server_negotiation.cc:407] Connection header received
0926 05:55:51.285450 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.285451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.285461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.285490 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.285945 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52930: BlockingRecv error: recv got EOF from 127.0.0.1:52930 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:55:51.288774 15523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:51.287880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.24.255.61:44759)
0926 05:55:51.287919 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.287921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.288209 (+ 288us) server_negotiation.cc:407] Connection header received
0926 05:55:51.288325 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.288326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.288347 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.288377 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.288721 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57908: BlockingRecv error: recv got EOF from 127.0.0.1:57908 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 05:55:51.292145 15523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:51.291248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.24.255.61:44759)
0926 05:55:51.291285 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.291286 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.291555 (+ 269us) server_negotiation.cc:407] Connection header received
0926 05:55:51.291661 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.291662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.291673 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.291703 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.292077 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57916: BlockingRecv error: recv got EOF from 127.0.0.1:57916 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 05:55:51.296257 15523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:51.295427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.24.255.61:44759)
0926 05:55:51.295467 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.295469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.295718 (+ 249us) server_negotiation.cc:407] Connection header received
0926 05:55:51.295813 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.295815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.295825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.295853 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.296204 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57924: BlockingRecv error: recv got EOF from 127.0.0.1:57924 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 05:55:51.302692 15523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:51.301633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.24.255.61:44759)
0926 05:55:51.301677 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.301680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.301970 (+ 290us) server_negotiation.cc:407] Connection header received
0926 05:55:51.302077 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.302078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.302089 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.302199 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.302630 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57928: BlockingRecv error: recv got EOF from 127.0.0.1:57928 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 05:55:51.313118 15523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:51.312175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.24.255.61:44759)
0926 05:55:51.312231 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.312233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.312531 (+ 298us) server_negotiation.cc:407] Connection header received
0926 05:55:51.312635 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.312636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.312657 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.312689 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.313055 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57932: BlockingRecv error: recv got EOF from 127.0.0.1:57932 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 05:55:51.331389 15523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:51.330549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.24.255.61:44759)
0926 05:55:51.330591 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.330593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.330852 (+ 259us) server_negotiation.cc:407] Connection header received
0926 05:55:51.330945 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.330947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.330958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.330986 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.331325 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57944: BlockingRecv error: recv got EOF from 127.0.0.1:57944 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 05:55:51.365438 15523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:51.364776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.24.255.61:44759)
0926 05:55:51.364823 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.364825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.364923 (+ 98us) server_negotiation.cc:407] Connection header received
0926 05:55:51.364980 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.364981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.364992 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.365040 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.365357 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57950: BlockingRecv error: recv got EOF from 127.0.0.1:57950 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 05:55:51.431588 15523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:51.430876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.24.255.61:44759)
0926 05:55:51.430934 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.430936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.431045 (+ 109us) server_negotiation.cc:407] Connection header received
0926 05:55:51.431126 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.431127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.431139 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.431184 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.431527 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57962: BlockingRecv error: recv got EOF from 127.0.0.1:57962 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 05:55:51.561600 15523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:51.560923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57976 (local address 127.24.255.61:44759)
0926 05:55:51.560992 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.560994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.561008 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:55:51.561109 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.561110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.561122 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.561173 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.561495 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57976: BlockingRecv error: recv got EOF from 127.0.0.1:57976 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 05:55:51.819545 15523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:51.818978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.24.255.61:44759)
0926 05:55:51.819046 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:51.819048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:51.819056 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:55:51.819087 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:51.819088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:51.819100 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:51.819154 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:51.819493 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57980: BlockingRecv error: recv got EOF from 127.0.0.1:57980 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:55:52.182062 15626 catalog_manager.cc:1259] Loaded cluster ID: fccc273487714a4e87488339d1742781
I20260926 05:55:52.182159 15626 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2: loading cluster ID for follower catalog manager: success
I20260926 05:55:52.182953 15626 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2: acquiring CA information for follower catalog manager: success
I20260926 05:55:52.183234 15626 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53342f81df484088a5bf786ca3694cb2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:55:52.252849 15654 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d00606e9f96443eda5038abca15cd49d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:55:52.335974 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:52.333166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.24.255.61:44759)
0926 05:55:52.333271 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:52.333273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:52.333281 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:55:52.333309 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:52.333309 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:52.333360 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:52.333446 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:52.333663 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:52.334229 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:52.335133 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:52.335336 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:52.335399 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:52.335583 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:52.335720 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:52.335795 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:52.335875 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 05:55:52.335902 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260926 05:55:52.340476 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:52.338085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.24.255.60:36759)
0926 05:55:52.338212 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:52.338214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:52.338221 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:55:52.338258 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:52.338259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:52.338297 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:52.338376 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:52.338539 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:52.339042 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:52.339772 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:52.339952 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:52.340010 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:52.340150 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:52.340259 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:52.340315 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:52.340388 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 05:55:52.340406 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":85,"threads_started":1}
I20260926 05:55:52.345397 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:52.342306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52946 (local address 127.24.255.60:36759)
0926 05:55:52.342427 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:52.342429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:52.342532 (+ 103us) server_negotiation.cc:407] Connection header received
0926 05:55:52.342641 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:52.342642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:52.342654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:52.342706 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:52.343061 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:52.343768 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:52.344499 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:52.344821 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:52.344885 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:52.345031 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:52.345094 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:52.345139 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:52.345204 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 05:55:52.345229 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 05:55:52.345531 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:52.342353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.24.255.61:44759)
0926 05:55:52.342397 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:52.342399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:52.342502 (+ 103us) server_negotiation.cc:407] Connection header received
0926 05:55:52.342587 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:52.342588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:52.342599 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:52.342646 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:52.342844 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:52.343640 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:52.344647 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:52.344923 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:52.344974 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:52.345264 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:52.345299 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:52.345347 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:52.345437 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 05:55:52.345461 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:55:52.345535 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:52.342248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.24.255.62:42395)
0926 05:55:52.342368 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:52.342370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:52.342636 (+ 266us) server_negotiation.cc:407] Connection header received
0926 05:55:52.342751 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:52.342753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:52.342810 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:52.342887 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:52.343014 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:52.343533 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:52.344649 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:52.344883 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:52.344922 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:52.345196 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:52.345323 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:52.345386 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:52.345425 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 05:55:52.345453 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260926 05:55:52.347674 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 15376
I20260926 05:55:52.353091 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 15451
I20260926 05:55:52.357601 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 15526
2026-09-26T05:55:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2796 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:55:52 dist-test-slave-7fr8 krb5kdc[15742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:55:52 dist-test-slave-7fr8 krb5kdc[15742](info): set up 2 sockets
Sep 26 05:55:52 dist-test-slave-7fr8 krb5kdc[15742](info): commencing operation
krb5kdc: starting...
W20260926 05:55:53.330482 25596 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/test-user.keytab.
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.255.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
Entry for principal kudu/127.24.255.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab.
I20260926 05:55:53.451319 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:33937
--webserver_interface=127.24.255.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:33937,127.24.255.61:38641,127.24.255.60:45971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:55:53.534361 15755 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:55:53.534590 15755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:53.534606 15755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:53.534627 15755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:53.535887 15755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:53.535938 15755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:53.535952 15755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:53.535964 15755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:53.535974 15755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:53.538635 15755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:33937,127.24.255.61:38641,127.24.255.60:45971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.62.keytab
--principal=kudu/127.24.255.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.62:33937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.255.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:53.538972 15755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:53.539240 15755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:53.544147 15761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:53.544138 15763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:53.544203 15755 server_base.cc:1034] running on GCE node
W20260926 05:55:53.544143 15760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:53.544567 15755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:53.544883 15755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:53.546057 15755 hybrid_clock.cc:648] HybridClock initialized: now 1790402153546034 us; error 39 us; skew 500 ppm
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:53.548890 15755 init.cc:377] Logged in from keytab as kudu/127.24.255.62@KRBTEST.COM (short username kudu)
I20260926 05:55:53.549357 15755 webserver.cc:460] Webserver started at http://127.24.255.62:46109/ using document root <none> and password file <none>
I20260926 05:55:53.549582 15755 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:53.549660 15755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:53.549787 15755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:53.550900 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/data/instance:
uuid: "0fe549c924384f25bd4f76771976e5a7"
format_stamp: "Formatted at 2026-09-26 05:55:53 on dist-test-slave-7fr8"
server_key: "abc9229d91c7d51788cb8872dd47a161"
server_key_iv: "8b5ee63b844b31317e83651364947b2f"
server_key_version: "encryptionkey@0"
I20260926 05:55:53.551239 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/wal/instance:
uuid: "0fe549c924384f25bd4f76771976e5a7"
format_stamp: "Formatted at 2026-09-26 05:55:53 on dist-test-slave-7fr8"
server_key: "abc9229d91c7d51788cb8872dd47a161"
server_key_iv: "8b5ee63b844b31317e83651364947b2f"
server_key_version: "encryptionkey@0"
I20260926 05:55:53.552853 15755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:55:53.553705 15770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:53.553898 15755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:55:53.553987 15755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/wal
uuid: "0fe549c924384f25bd4f76771976e5a7"
format_stamp: "Formatted at 2026-09-26 05:55:53 on dist-test-slave-7fr8"
server_key: "abc9229d91c7d51788cb8872dd47a161"
server_key_iv: "8b5ee63b844b31317e83651364947b2f"
server_key_version: "encryptionkey@0"
I20260926 05:55:53.554052 15755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:53.571712 15755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:53.573743 15755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:53.573884 15755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:53.577984 15755 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.62:33937
I20260926 05:55:53.578045 15822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.62:33937 every 8 connection(s)
I20260926 05:55:53.578394 15755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/data/info.pb
I20260926 05:55:53.579373 15823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:53.581197 15823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:55:53.586499 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 15755
I20260926 05:55:53.586603 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-0/wal/instance
I20260926 05:55:53.586819 25596 external_mini_cluster.cc:1428] Setting key 81e308b7bbedff3da2e1a258f76d8b4b
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
I20260926 05:55:53.590623 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.581949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.24.255.62:33937)
0926 05:55:53.582172 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.582181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.582285 (+ 104us) server_negotiation.cc:407] Connection header received
0926 05:55:53.582861 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.582863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.582881 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.582957 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.583504 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.584316 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.585140 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.585442 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.585541 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.587664 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.587671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.587671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.587679 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.589091 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.589424 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.589427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.589428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.589498 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.589688 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.589689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.589689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.589962 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.590038 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.590242 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.590359 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":70,"threads_started":1}
I20260926 05:55:53.590626 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.581865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:33937 (local address 127.0.0.1:48246)
0926 05:55:53.582158 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.582206 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:53.582715 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:53.582970 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:53.582973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:53.583261 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:53.583415 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.583420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.584401 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.584402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.585065 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.585066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.585533 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.585534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.585603 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.586166 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:53.586175 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:53.587578 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:53.589166 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.589172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.589175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.589357 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.589552 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.589553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.589553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.589637 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.590025 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:53.590031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:53.590117 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 05:55:53.590208 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 05:55:53.590431 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":81,"thread_start_us":93,"threads_started":1}
I20260926 05:55:53.591594 15823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:55:53.591990 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.591852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38641 (local address 127.0.0.1:34202)
0926 05:55:53.591895 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.591940 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 05:55:53.592113 15773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38641: connect: Connection refused (error 111)
I20260926 05:55:53.592526 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.592435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38641 (local address 127.0.0.1:34204)
0926 05:55:53.592474 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.592501 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 05:55:53.592693 15823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.61:38641: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38641: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.24.255.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.61.keytab.
Entry for principal kudu/127.24.255.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.61.keytab.
I20260926 05:55:53.613351 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.61:38641
--webserver_interface=127.24.255.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.61.keytab
--principal=kudu/127.24.255.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:33937,127.24.255.61:38641,127.24.255.60:45971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:55:53.642011 15823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } attempt: 1
I20260926 05:55:53.642647 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.642519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38641 (local address 127.0.0.1:34210)
0926 05:55:53.642578 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.642615 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:55:53.642963 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.642887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38641 (local address 127.0.0.1:34224)
0926 05:55:53.642921 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.642945 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 05:55:53.643047 15823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.61:38641: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38641: connect: Connection refused (error 111)
W20260926 05:55:53.695961 15830 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:55:53.696200 15830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:53.696225 15830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:53.696251 15830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:53.697441 15830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:53.697484 15830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:53.697503 15830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:53.697518 15830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:53.697535 15830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:53.699162 15830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:33937,127.24.255.61:38641,127.24.255.60:45971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.61.keytab
--principal=kudu/127.24.255.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.61:38641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.255.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:53.699429 15830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:53.699645 15830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:53.704393 15836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:53.704380 15838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:53.704362 15835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:53.704818 15830 server_base.cc:1034] running on GCE node
I20260926 05:55:53.705058 15830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:53.705381 15830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:53.706532 15830 hybrid_clock.cc:648] HybridClock initialized: now 1790402153706488 us; error 55 us; skew 500 ppm
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:53.709035 15830 init.cc:377] Logged in from keytab as kudu/127.24.255.61@KRBTEST.COM (short username kudu)
I20260926 05:55:53.709570 15830 webserver.cc:460] Webserver started at http://127.24.255.61:43873/ using document root <none> and password file <none>
I20260926 05:55:53.709775 15830 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:53.709824 15830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:53.709952 15830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:53.711062 15830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/data/instance:
uuid: "b9a414a274344eaca2ae3c6d497dd762"
format_stamp: "Formatted at 2026-09-26 05:55:53 on dist-test-slave-7fr8"
server_key: "11e51f8cac85913f03307fee7590f275"
server_key_iv: "33637f30cbaa703f7a2fc62b11f54d2f"
server_key_version: "encryptionkey@0"
I20260926 05:55:53.711196 15823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } attempt: 2
I20260926 05:55:53.711457 15830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/wal/instance:
uuid: "b9a414a274344eaca2ae3c6d497dd762"
format_stamp: "Formatted at 2026-09-26 05:55:53 on dist-test-slave-7fr8"
server_key: "11e51f8cac85913f03307fee7590f275"
server_key_iv: "33637f30cbaa703f7a2fc62b11f54d2f"
server_key_version: "encryptionkey@0"
I20260926 05:55:53.711632 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.711439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38641 (local address 127.0.0.1:34228)
0926 05:55:53.711544 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.711590 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 05:55:53.711939 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.711864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38641 (local address 127.0.0.1:34242)
0926 05:55:53.711901 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.711923 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 05:55:53.712028 15823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.61:38641: Network error: Client connection negotiation failed: client connection to 127.24.255.61:38641: connect: Connection refused (error 111)
I20260926 05:55:53.712926 15830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:55:53.713687 15845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:53.713876 15830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:53.714000 15830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/wal
uuid: "b9a414a274344eaca2ae3c6d497dd762"
format_stamp: "Formatted at 2026-09-26 05:55:53 on dist-test-slave-7fr8"
server_key: "11e51f8cac85913f03307fee7590f275"
server_key_iv: "33637f30cbaa703f7a2fc62b11f54d2f"
server_key_version: "encryptionkey@0"
I20260926 05:55:53.714063 15830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:53.739043 15830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:53.741283 15830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:53.741400 15830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:53.745452 15830 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.61:38641
I20260926 05:55:53.745522 15897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.61:38641 every 8 connection(s)
I20260926 05:55:53.745831 15830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/data/info.pb
I20260926 05:55:53.746814 15898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:53.748379 15898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:55:53.748602 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 15830
I20260926 05:55:53.748713 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-1/wal/instance
I20260926 05:55:53.748917 25596 external_mini_cluster.cc:1428] Setting key 3bcf35a686afbb15291a55c45fbad85f
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.61@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.255.60@KRBTEST.COM; defaulting to no policy
I20260926 05:55:53.758085 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.749125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.24.255.62:33937)
0926 05:55:53.749171 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.749173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.749456 (+ 283us) server_negotiation.cc:407] Connection header received
0926 05:55:53.749991 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.749993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.750009 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.750056 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.750777 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.751268 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.752242 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.752444 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.752501 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.755070 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.755072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.755072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.755081 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.756532 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.756978 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.756979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.756979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.757036 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.757357 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.757358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.757358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.757528 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.757590 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.757903 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.757993 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:55:53.758258 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.749124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:33937 (local address 127.0.0.1:48254)
0926 05:55:53.749332 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.749373 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:53.749887 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:53.750166 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:53.750169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:53.750429 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:53.750677 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.750682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.751352 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.751353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.752151 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.752153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.752539 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.752540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.752654 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.753306 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:53.753315 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:53.754946 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:53.756613 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.756621 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.756623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.756910 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.757119 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.757120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.757120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.757254 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.757601 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:53.757607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:53.757745 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 05:55:53.757847 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 05:55:53.758025 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":97,"threads_started":1}
I20260926 05:55:53.758800 15898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.255.60@KRBTEST.COM" created.
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.61@KRBTEST.COM for kudu/127.24.255.61@KRBTEST.COM
I20260926 05:55:53.767539 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.759158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38641 (local address 127.0.0.1:34248)
0926 05:55:53.759239 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.759257 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:53.759353 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:53.759705 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:53.759707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:53.759852 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:53.759923 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.759925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.760884 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.760884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.761378 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.761379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.761841 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.761842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.761905 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.762362 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:53.762367 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:53.764376 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:53.766490 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.766491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.766491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.766686 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.766901 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.766901 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.766901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.766953 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.767287 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:53.767288 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:53.767330 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 05:55:53.767385 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 05:55:53.767465 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 05:55:53.767762 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.759300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.24.255.61:38641)
0926 05:55:53.759470 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.759475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.759499 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:55:53.759525 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.759528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.759543 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.759631 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.760029 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.760775 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.761448 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.761764 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.761833 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.764479 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.764490 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.764491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.764499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.766375 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.766775 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.766778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.766779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.766840 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.767007 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.767008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.767008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.767214 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.767303 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.767440 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.767520 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260926 05:55:53.768415 15898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:55:53.768797 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.768658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:45971 (local address 127.0.0.1:43060)
0926 05:55:53.768720 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.768772 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 05:55:53.768925 15846 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
I20260926 05:55:53.769336 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.769250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:45971 (local address 127.0.0.1:43074)
0926 05:55:53.769293 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.769317 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 05:55:53.769421 15898 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:45971: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.255.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.60.keytab.
Entry for principal kudu/127.24.255.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.60.keytab.
I20260926 05:55:53.776855 25596 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGUnhcE/build/release/bin/kudu
/tmp/dist-test-taskGUnhcE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.60:45971
--webserver_interface=127.24.255.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.60.keytab
--principal=kudu/127.24.255.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.255.20:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.255.62:33937,127.24.255.61:38641,127.24.255.60:45971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:55:53.803285 15823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } attempt: 3
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for kudu/127.24.255.61@KRBTEST.COM
I20260926 05:55:53.811666 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.803681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38641 (local address 127.0.0.1:34254)
0926 05:55:53.803746 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.803759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:53.803809 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:53.804034 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:53.804035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:53.804191 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:53.804260 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.804262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.805248 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.805249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.805800 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.805801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.806428 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.806429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.806504 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.806989 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:53.806994 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:53.808503 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:53.810560 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.810561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.810561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.810816 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.811045 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.811046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.811047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.811113 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.811375 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:53.811376 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:53.811421 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 05:55:53.811469 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:55:53.811562 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:55:53.811692 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.803798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.24.255.61:38641)
0926 05:55:53.803859 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.803861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.803872 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:55:53.803903 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.803904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.803917 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.803998 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.804340 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.805136 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.805927 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.806313 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.806417 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.808623 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.808625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.808625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.808636 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.810453 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.810892 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.810894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.810894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.810961 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.811177 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.811177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.811178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.811320 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.811386 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.811515 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.811601 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:55:53.812101 15823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:55:53.812443 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.812351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:45971 (local address 127.0.0.1:43090)
0926 05:55:53.812395 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.812422 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 05:55:53.812748 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.812675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:45971 (local address 127.0.0.1:43098)
0926 05:55:53.812712 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.812734 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 05:55:53.812856 15823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:45971: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
I20260926 05:55:53.818651 15898 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } attempt: 1
I20260926 05:55:53.819298 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.819122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:45971 (local address 127.0.0.1:43108)
0926 05:55:53.819197 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.819262 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 05:55:53.819890 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.819680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:45971 (local address 127.0.0.1:43116)
0926 05:55:53.819741 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.819797 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 05:55:53.820158 15898 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:45971: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
I20260926 05:55:53.844044 15823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } attempt: 1
I20260926 05:55:53.844632 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.844485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:45971 (local address 127.0.0.1:43124)
0926 05:55:53.844556 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.844601 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 05:55:53.845047 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.844898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:45971 (local address 127.0.0.1:43130)
0926 05:55:53.844965 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.845003 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 05:55:53.845255 15823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:45971: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
W20260926 05:55:53.865940 15905 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:55:53.866358 15905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:55:53.866379 15905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:55:53.866401 15905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:55:53.867749 15905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:55:53.867805 15905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:55:53.867818 15905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:55:53.867830 15905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:55:53.867841 15905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:55:53.869511 15905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.255.20:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.255.62:33937,127.24.255.61:38641,127.24.255.60:45971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/krb5kdc/kudu_127.24.255.60.keytab
--principal=kudu/127.24.255.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.255.60:45971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.255.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:55:53.869761 15905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:55:53.869949 15905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:55:53.874873 15913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:53.874878 15910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:55:53.874853 15911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:55:53.875413 15905 server_base.cc:1034] running on GCE node
I20260926 05:55:53.875682 15905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:55:53.876001 15905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:55:53.877188 15905 hybrid_clock.cc:648] HybridClock initialized: now 1790402153877167 us; error 38 us; skew 500 ppm
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:55:53.880028 15905 init.cc:377] Logged in from keytab as kudu/127.24.255.60@KRBTEST.COM (short username kudu)
I20260926 05:55:53.880491 15905 webserver.cc:460] Webserver started at http://127.24.255.60:44533/ using document root <none> and password file <none>
I20260926 05:55:53.880703 15905 fs_manager.cc:359] Metadata directory not provided
I20260926 05:55:53.880748 15905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:55:53.880870 15905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:55:53.881981 15905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/data/instance:
uuid: "2ec2892815b34d22a112b667213f077b"
format_stamp: "Formatted at 2026-09-26 05:55:53 on dist-test-slave-7fr8"
server_key: "4705d0631ad156f48e4cf0a5d8d8b08a"
server_key_iv: "57b58fb5420300e4c391081984f02b6e"
server_key_version: "encryptionkey@0"
I20260926 05:55:53.882364 15905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/wal/instance:
uuid: "2ec2892815b34d22a112b667213f077b"
format_stamp: "Formatted at 2026-09-26 05:55:53 on dist-test-slave-7fr8"
server_key: "4705d0631ad156f48e4cf0a5d8d8b08a"
server_key_iv: "57b58fb5420300e4c391081984f02b6e"
server_key_version: "encryptionkey@0"
I20260926 05:55:53.883903 15905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:53.884799 15920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:55:53.885082 15905 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:55:53.885167 15905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/data,/tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/wal
uuid: "2ec2892815b34d22a112b667213f077b"
format_stamp: "Formatted at 2026-09-26 05:55:53 on dist-test-slave-7fr8"
server_key: "4705d0631ad156f48e4cf0a5d8d8b08a"
server_key_iv: "57b58fb5420300e4c391081984f02b6e"
server_key_version: "encryptionkey@0"
I20260926 05:55:53.885257 15905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:55:53.888353 15898 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } attempt: 2
I20260926 05:55:53.888875 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.888764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:45971 (local address 127.0.0.1:43140)
0926 05:55:53.888810 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.888844 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 05:55:53.889206 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.889116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:45971 (local address 127.0.0.1:43148)
0926 05:55:53.889165 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.889188 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 05:55:53.889283 15898 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.255.60:45971: Network error: Client connection negotiation failed: client connection to 127.24.255.60:45971: connect: Connection refused (error 111)
I20260926 05:55:53.905659 15905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:55:53.908754 15905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:55:53.909016 15905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:55:53.913889 15905 rpc_server.cc:307] RPC server started. Bound to: 127.24.255.60:45971
I20260926 05:55:53.913924 15972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.255.60:45971 every 8 connection(s)
I20260926 05:55:53.914366 15905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/data/info.pb
I20260926 05:55:53.915371 15973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:55:53.917007 15973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:55:53.920425 15823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } attempt: 2
I20260926 05:55:53.922683 25596 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGUnhcE/build/release/bin/kudu as pid 15905
I20260926 05:55:53.922823 25596 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGUnhcE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401881171543-25596-0/minicluster-data/master-2/wal/instance
I20260926 05:55:53.923043 25596 external_mini_cluster.cc:1428] Setting key 6d2ffa4930fb7cdea466da8ff2f29aa0
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.60@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.62@KRBTEST.COM for kudu/127.24.255.60@KRBTEST.COM
I20260926 05:55:53.929066 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.918118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.24.255.62:33937)
0926 05:55:53.918204 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.918207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.918742 (+ 535us) server_negotiation.cc:407] Connection header received
0926 05:55:53.919444 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.919446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.919462 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.919515 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.920192 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.920718 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.921855 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.922055 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.922153 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.925369 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.925371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.925371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.925378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.927249 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.927940 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.927941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.927942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.928007 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.928307 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.928308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.928308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.928502 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.928560 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.928880 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.928968 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:55:53.929250 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.920750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:45971 (local address 127.0.0.1:43154)
0926 05:55:53.920808 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.920820 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:53.920845 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:53.921384 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:53.921386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:53.921572 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:53.921628 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.921630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.922705 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.922707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.923391 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.923396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.924006 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.924007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.924076 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.924508 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:53.924513 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:53.925817 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:53.928166 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.928167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.928167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.928371 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.928595 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.928596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.928596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.928653 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.928975 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:53.928976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:53.929019 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 05:55:53.929079 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:55:53.929170 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 05:55:53.929272 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.918144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.62:33937 (local address 127.0.0.1:48264)
0926 05:55:53.918546 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.918601 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:53.919269 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:53.919599 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:53.919603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:53.919887 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:53.920098 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.920101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.920807 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.920808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.921741 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.921747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.922191 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.922192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.922325 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.923627 (+ 1302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:53.923639 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:53.925280 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:53.927544 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.927552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.927555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.927829 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.928080 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.928081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.928081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.928236 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.928567 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:53.928574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:53.928726 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 05:55:53.928841 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 05:55:53.929030 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":242,"threads_started":1}
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.60@KRBTEST.COM
I20260926 05:55:53.929555 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.920796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43154 (local address 127.24.255.60:45971)
0926 05:55:53.921071 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.921077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.921104 (+ 27us) server_negotiation.cc:407] Connection header received
0926 05:55:53.921134 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.921136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.921157 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.921261 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.921707 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.922600 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.923508 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.923915 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.923982 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.925917 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.925926 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.925927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.925934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.928072 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.928452 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.928455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.928455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.928519 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.928715 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.928716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.928717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.928876 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.928944 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.929383 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.929461 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":130,"threads_started":1}
I20260926 05:55:53.929919 15973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:55:53.931545 15823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7: Bootstrap starting.
I20260926 05:55:53.932281 15823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:53.932590 15823 log.cc:824] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:53.933630 15823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7: No bootstrap required, opened a new log
I20260926 05:55:53.934259 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.923490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.24.255.60:45971)
0926 05:55:53.923705 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.923707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.924162 (+ 455us) server_negotiation.cc:407] Connection header received
0926 05:55:53.924206 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.924207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.924218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.924286 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.925029 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.925823 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.926759 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.926985 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.927065 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.930565 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.930566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.930567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.930573 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.932641 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.933019 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.933021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.933021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.933084 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.933284 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.933285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.933285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.933424 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.933580 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.934038 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.934138 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":140,"threads_started":1}
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.60@KRBTEST.COM for kudu/127.24.255.61@KRBTEST.COM
I20260926 05:55:53.935565 15823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } }
I20260926 05:55:53.935706 15823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe549c924384f25bd4f76771976e5a7, State: Initialized, Role: FOLLOWER
I20260926 05:55:53.935792 15823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } }
I20260926 05:55:53.936137 15980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } } }
I20260926 05:55:53.936252 15980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:55:53.936257 15823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:53.937731 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.930198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.61:38641 (local address 127.0.0.1:34270)
0926 05:55:53.930241 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.930250 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:53.930282 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:53.930428 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:53.930429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:53.930563 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:53.930621 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.930622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.931444 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.931445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.932410 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.932414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.932857 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.932858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.932932 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.933286 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:53.933290 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:53.934737 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:53.936640 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.936641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.936642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.936857 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.937069 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.937069 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.937070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.937135 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.937421 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:53.937423 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:53.937471 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 05:55:53.937534 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 05:55:53.937639 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 05:55:53.938185 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.930268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.24.255.61:38641)
0926 05:55:53.930312 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.930314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.930322 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:55:53.930347 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.930348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.930358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.930420 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.930701 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.931353 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.932503 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.932784 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.932848 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.934811 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.934812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.934813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.934820 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.936550 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.936927 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.936928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.936928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.936997 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.937190 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.937191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.937192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.937331 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.937391 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.937965 (+ 574us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.938047 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:55:53.938515 15973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } has no permanent_uuid. Determining permanent_uuid...
W20260926 05:55:53.938231 15991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:55:53.938678 15991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.62@KRBTEST.COM
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.60@KRBTEST.COM for kudu/127.24.255.60@KRBTEST.COM
I20260926 05:55:53.944087 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.934901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.24.255.62:33937)
0926 05:55:53.934940 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.934941 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.934949 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:55:53.934990 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.934991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.935000 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.935043 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.935709 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.936415 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.937312 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.937598 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.937659 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.940547 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.940549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.940550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.940558 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.942382 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.942785 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.942786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.942786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.942847 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.943047 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.943048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.943048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.943218 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.943395 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.943921 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.944002 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:55:53.941165 15755 master_runner.cc:186] Error getting master registration for 127.24.255.61:38641: OK, instance_id { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" instance_seqno: 1790402153744275 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:55:53.945309 15755 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:55:53.946409 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.938762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:45971 (local address 127.0.0.1:43172)
0926 05:55:53.938794 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.938805 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:53.938841 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:53.938992 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:53.938993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:53.939107 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:53.939140 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.939141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.940060 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.940061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.940892 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.940897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.941286 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.941287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.941339 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.941881 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:53.941886 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:53.943400 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:53.945206 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.945207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.945207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.945399 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.945612 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.945613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.945614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.945695 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.946062 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:53.946063 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:53.946148 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 05:55:53.946208 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:55:53.946309 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260926 05:55:53.946750 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.938826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43172 (local address 127.24.255.60:45971)
0926 05:55:53.938884 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.938886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.938895 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:55:53.938923 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.938924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.938937 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.938992 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.939204 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.939874 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.940997 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.941217 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.941274 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.943492 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.943493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.943494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.943515 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.945122 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.945468 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.945469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.945469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.945533 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.945771 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.945772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.945772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.945973 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.946038 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.946593 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.946672 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:55:53.948174 15973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b: Bootstrap starting.
I20260926 05:55:53.949332 15973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:53.949659 15973 log.cc:824] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:53.950425 15973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b: No bootstrap required, opened a new log
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.255.61@KRBTEST.COM
I20260926 05:55:53.952987 15973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } }
I20260926 05:55:53.953131 15973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec2892815b34d22a112b667213f077b, State: Initialized, Role: FOLLOWER
I20260926 05:55:53.953217 15973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } }
I20260926 05:55:53.953478 15994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } } }
I20260926 05:55:53.953557 15973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:53.953624 15994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:55:53.954353 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.945052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.24.255.61:38641)
0926 05:55:53.945104 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.945106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.945209 (+ 103us) server_negotiation.cc:407] Connection header received
0926 05:55:53.945235 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.945236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.945247 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.945286 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.946015 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.946676 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.947690 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.948007 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.948081 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.951036 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.951039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.951039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.951048 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.952737 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.953209 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.953210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.953211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.953276 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.953463 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.953463 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.953464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.953602 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.953745 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.954171 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.954255 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:55:53.955852 16005 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:55:53.955909 16005 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:55:53.956350 25596 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:55:53.957290 15797 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0fe549c924384f25bd4f76771976e5a7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48274
I20260926 05:55:53.957412 15797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 05:55:53.957456 15797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:53.958832 15797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } }
I20260926 05:55:53.959074 15797 leader_election.cc:288] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [CANDIDATE]: Term 1 election: Requested vote from peers b9a414a274344eaca2ae3c6d497dd762 (127.24.255.61:38641), 2ec2892815b34d22a112b667213f077b (127.24.255.60:45971)
I20260926 05:55:53.956969 15905 master_runner.cc:186] Error getting master registration for 127.24.255.61:38641: OK, instance_id { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" instance_seqno: 1790402153744275 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:55:53.959280 15905 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:55:53.959376 15947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fe549c924384f25bd4f76771976e5a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2ec2892815b34d22a112b667213f077b"
I20260926 05:55:53.959494 15947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:53.960408 15947 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe549c924384f25bd4f76771976e5a7 in term 1.
I20260926 05:55:53.960632 15771 leader_election.cc:302] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe549c924384f25bd4f76771976e5a7, 2ec2892815b34d22a112b667213f077b; no voters:
I20260926 05:55:53.960788 15980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:55:53.960939 15980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [term 1 LEADER]: Becoming Leader. State: Replica: 0fe549c924384f25bd4f76771976e5a7, State: Running, Role: LEADER
I20260926 05:55:53.961040 15980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } }
I20260926 05:55:53.961530 16013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fe549c924384f25bd4f76771976e5a7. Latest consensus state: current_term: 1 leader_uuid: "0fe549c924384f25bd4f76771976e5a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } } }
I20260926 05:55:53.961675 16013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:53.962519 16016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:55:53.962775 16016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:55:53.964799 15947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b [term 1 FOLLOWER]: Refusing update from remote peer 0fe549c924384f25bd4f76771976e5a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:53.965269 15980 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:53.966825 15994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fe549c924384f25bd4f76771976e5a7. Latest consensus state: current_term: 1 leader_uuid: "0fe549c924384f25bd4f76771976e5a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } } }
I20260926 05:55:53.966941 15994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:55:53.969566 15994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fe549c924384f25bd4f76771976e5a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } } }
I20260926 05:55:53.969655 15994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:55:53.968847 16017 mvcc.cc:205] Tried to move back new op lower bound from 7333487222635511808 to 7333487222625136640. Current Snapshot: MvccSnapshot[applied={T|T < 7333487222635511808}]
I20260926 05:55:53.970269 15980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fe549c924384f25bd4f76771976e5a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } } }
I20260926 05:55:53.970358 15980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [sys.catalog]: This master's current role is: LEADER
I20260926 05:55:53.971125 16018 mvcc.cc:205] Tried to move back new op lower bound from 7333487222635511808 to 7333487222625136640. Current Snapshot: MvccSnapshot[applied={T|T < 7333487222635511808}]
I20260926 05:55:53.972069 16016 catalog_manager.cc:1347] Generated new cluster ID: 443527292bf146e48e733c68b858a673
I20260926 05:55:53.972118 16016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:55:53.972241 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.960613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.24.255.61:38641)
0926 05:55:53.960658 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.960660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.961417 (+ 757us) server_negotiation.cc:407] Connection header received
0926 05:55:53.961464 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.961465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.961476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.961505 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.962716 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.963377 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.964914 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.965629 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.965673 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.968354 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.968355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.968356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.968377 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.970065 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.970914 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.970915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.970915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.970982 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.971528 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.971530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.971530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.971674 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.971765 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.972075 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.972160 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:55:53.978538 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.961552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43178 (local address 127.24.255.60:45971)
0926 05:55:53.961592 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.961594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.961602 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:55:53.961623 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.961624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.961632 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.961688 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.962538 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.963289 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.966180 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.966491 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.966532 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.970189 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.970190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.970191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.970199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.972045 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.972513 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.972514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.972515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.972571 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.973582 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.973583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.973583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.973740 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.973833 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.978289 (+ 4456us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.978424 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:55:53.980408 15898 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } attempt: 3
Sep 26 05:55:53 dist-test-slave-7fr8 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790402153, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.255.61@KRBTEST.COM for kudu/127.24.255.60@KRBTEST.COM
I20260926 05:55:53.990031 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.981736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.24.255.60:45971)
0926 05:55:53.981892 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.981894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.981900 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:55:53.981920 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.981920 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.981929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.981969 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.982353 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.983375 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.984419 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.984709 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.984764 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.986828 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.986830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.986830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.986837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.988572 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.988968 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.988970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.988970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.989047 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.989280 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.989281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.989281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.989427 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.989496 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.989854 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.989931 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260926 05:55:53.990041 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.981655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.255.60:45971 (local address 127.0.0.1:43192)
0926 05:55:53.981776 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 05:55:53.981793 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 05:55:53.981829 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:55:53.982011 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:55:53.982011 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:55:53.982223 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:55:53.982274 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.982276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.983485 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.983486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.984328 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:55:53.984331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.984837 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.984839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:55:53.984894 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.985272 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:55:53.985277 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:55:53.986732 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:55:53.988677 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.988678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.988678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.988900 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.989151 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:55:53.989151 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:55:53.989151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:55:53.989225 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:55:53.989691 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:55:53.989692 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:55:53.989754 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 05:55:53.989819 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 05:55:53.989931 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 05:55:53.990052 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:53.961734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.24.255.62:33937)
0926 05:55:53.961770 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:53.961772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:53.961778 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:55:53.961796 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:53.961797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:53.961806 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:53.961850 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:53.966881 (+ 5031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.967565 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.970486 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:53.970827 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:53.970867 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:53.979268 (+ 8401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:55:53.979271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:55:53.979271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:55:53.979281 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:55:53.981413 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.987661 (+ 6248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.987663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.987664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.987733 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:55:53.989360 (+ 1627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:55:53.989361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:55:53.989361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:55:53.989503 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:55:53.989584 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:53.989861 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 05:55:53.989953 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:55:53.990525 16016 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:55:53.991407 16016 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:55:53.992213 15898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762: Bootstrap starting.
I20260926 05:55:53.993346 15898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762: Neither blocks nor log segments found. Creating new log.
I20260926 05:55:53.993721 15898 log.cc:824] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762: Log is configured to *not* fsync() on all Append() calls
I20260926 05:55:53.995172 15898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762: No bootstrap required, opened a new log
I20260926 05:55:53.997682 15898 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } }
I20260926 05:55:53.997874 15898 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a414a274344eaca2ae3c6d497dd762, State: Initialized, Role: FOLLOWER
I20260926 05:55:53.998559 16016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7: Generated new TSK 0
I20260926 05:55:53.998801 16016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:55:53.997956 15898 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } }
I20260926 05:55:53.999627 15898 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:55:53.999711 15871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:55:53.999675 15872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fe549c924384f25bd4f76771976e5a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b9a414a274344eaca2ae3c6d497dd762"
I20260926 05:55:54.000656 15871 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762 [term 1 FOLLOWER]: Refusing update from remote peer 0fe549c924384f25bd4f76771976e5a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:55:54.000998 15980 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:55:54.001690 15872 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe549c924384f25bd4f76771976e5a7 in term 1.
I20260926 05:55:54.003283 16036 mvcc.cc:205] Tried to move back new op lower bound from 7333487222773121024 to 7333487222625136640. Current Snapshot: MvccSnapshot[applied={T|T < 7333487222635511808}]
W20260926 05:55:54.004133 16041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 05:55:54.004383 16035 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fe549c924384f25bd4f76771976e5a7. Latest consensus state: current_term: 1 leader_uuid: "0fe549c924384f25bd4f76771976e5a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } } }
I20260926 05:55:54.004779 16035 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:55:54.004844 16035 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fe549c924384f25bd4f76771976e5a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } } }
I20260926 05:55:54.004904 16035 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:55:54.004602 16038 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fe549c924384f25bd4f76771976e5a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } } }
I20260926 05:55:54.004969 16038 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:55:54.005010 16041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 05:55:54.026618 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:54.024854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.24.255.61:38641)
0926 05:55:54.024905 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:54.024907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:54.025866 (+ 959us) server_negotiation.cc:407] Connection header received
0926 05:55:54.025968 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:54.025969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:54.025980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:54.026009 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:54.026512 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34284: BlockingRecv error: recv got EOF from 127.0.0.1:34284 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 05:55:54.027376 15826 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:54.025281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.24.255.62:33937)
0926 05:55:54.025320 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:54.025322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:54.026879 (+ 1557us) server_negotiation.cc:407] Connection header received
0926 05:55:54.026963 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:54.026964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:54.026974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:54.026999 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:54.027317 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48302: BlockingRecv error: recv got EOF from 127.0.0.1:48302 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 05:55:54.028085 15978 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:54.025577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.24.255.60:45971)
0926 05:55:54.025615 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:54.025617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:54.027560 (+ 1943us) server_negotiation.cc:407] Connection header received
0926 05:55:54.027687 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:54.027688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:54.027698 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:54.027725 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:54.028026 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43208: BlockingRecv error: recv got EOF from 127.0.0.1:43208 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 05:55:54.030690 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:54.029350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.24.255.61:38641)
0926 05:55:54.029387 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:54.029389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:54.030031 (+ 642us) server_negotiation.cc:407] Connection header received
0926 05:55:54.030230 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:54.030231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:54.030253 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:54.030282 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:54.030609 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34300: BlockingRecv error: recv got EOF from 127.0.0.1:34300 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 05:55:54.033849 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:54.032996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.24.255.61:38641)
0926 05:55:54.033033 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:54.033035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:54.033295 (+ 260us) server_negotiation.cc:407] Connection header received
0926 05:55:54.033388 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:54.033389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:54.033399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:54.033427 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:54.033791 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34314: BlockingRecv error: recv got EOF from 127.0.0.1:34314 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 05:55:54.038134 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:54.037089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.24.255.61:38641)
0926 05:55:54.037168 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:54.037170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:54.037475 (+ 305us) server_negotiation.cc:407] Connection header received
0926 05:55:54.037576 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:54.037577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:54.037589 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:54.037619 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:54.038044 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34320: BlockingRecv error: recv got EOF from 127.0.0.1:34320 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 05:55:54.044320 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:54.043408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.24.255.61:38641)
0926 05:55:54.043453 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:54.043455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:54.043739 (+ 284us) server_negotiation.cc:407] Connection header received
0926 05:55:54.043839 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:54.043840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:54.043852 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:54.043880 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:54.044259 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34324: BlockingRecv error: recv got EOF from 127.0.0.1:34324 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:55:54.055052 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:54.053869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.24.255.61:38641)
0926 05:55:54.053915 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:54.053917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:54.054437 (+ 520us) server_negotiation.cc:407] Connection header received
0926 05:55:54.054544 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:54.054545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:54.054559 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:54.054588 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:54.054991 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34326: BlockingRecv error: recv got EOF from 127.0.0.1:34326 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:55:54.074194 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:54.073121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.24.255.61:38641)
0926 05:55:54.073165 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:54.073167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:54.073494 (+ 327us) server_negotiation.cc:407] Connection header received
0926 05:55:54.073604 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:54.073605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:54.073617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:54.073647 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:54.074131 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34342: BlockingRecv error: recv got EOF from 127.0.0.1:34342 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:55:54.110364 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:54.108753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34356 (local address 127.24.255.61:38641)
0926 05:55:54.108818 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:54.108820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:54.109288 (+ 468us) server_negotiation.cc:407] Connection header received
0926 05:55:54.109672 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:54.109674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:54.109691 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:54.109723 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:54.110293 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34356: BlockingRecv error: recv got EOF from 127.0.0.1:34356 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 05:55:54.176776 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:54.175842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.24.255.61:38641)
0926 05:55:54.175899 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:54.175901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:54.176127 (+ 226us) server_negotiation.cc:407] Connection header received
0926 05:55:54.176239 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:54.176240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:54.176254 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:54.176285 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:54.176711 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34358: BlockingRecv error: recv got EOF from 127.0.0.1:34358 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 05:55:54.307153 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:54.306371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34372 (local address 127.24.255.61:38641)
0926 05:55:54.306470 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:54.306472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:54.306517 (+ 45us) server_negotiation.cc:407] Connection header received
0926 05:55:54.306604 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:54.306605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:54.306616 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:54.306676 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:54.307088 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34372: BlockingRecv error: recv got EOF from 127.0.0.1:34372 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 05:55:54.415915 15980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fe549c924384f25bd4f76771976e5a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe549c924384f25bd4f76771976e5a7" member_type: VOTER last_known_addr { host: "127.24.255.62" port: 33937 } } peers { permanent_uuid: "b9a414a274344eaca2ae3c6d497dd762" member_type: VOTER last_known_addr { host: "127.24.255.61" port: 38641 } } peers { permanent_uuid: "2ec2892815b34d22a112b667213f077b" member_type: VOTER last_known_addr { host: "127.24.255.60" port: 45971 } } }
I20260926 05:55:54.416071 15980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe549c924384f25bd4f76771976e5a7 [sys.catalog]: This master's current role is: LEADER
W20260926 05:55:54.565263 15902 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:55:54.564546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.24.255.61:38641)
0926 05:55:54.564636 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:54.564639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:54.564717 (+ 78us) server_negotiation.cc:407] Connection header received
0926 05:55:54.564797 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:54.564798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:54.564810 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:54.564859 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:55:54.565198 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34386: BlockingRecv error: recv got EOF from 127.0.0.1:34386 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 05:55:54.956414 16005 catalog_manager.cc:1259] Loaded cluster ID: 443527292bf146e48e733c68b858a673
I20260926 05:55:54.956494 16005 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b: loading cluster ID for follower catalog manager: success
I20260926 05:55:54.957412 16005 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b: acquiring CA information for follower catalog manager: success
I20260926 05:55:54.957719 16005 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ec2892815b34d22a112b667213f077b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:55:55.005522 16041 catalog_manager.cc:1259] Loaded cluster ID: 443527292bf146e48e733c68b858a673
I20260926 05:55:55.005601 16041 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762: loading cluster ID for follower catalog manager: success
I20260926 05:55:55.006490 16041 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762: acquiring CA information for follower catalog manager: success
I20260926 05:55:55.006850 16041 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b9a414a274344eaca2ae3c6d497dd762: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:55:55.081638 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:55.078506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.24.255.61:38641)
0926 05:55:55.078615 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:55.078616 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:55.078627 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:55:55.078656 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:55.078657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:55.078709 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:55.078799 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:55.079039 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:55.079568 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:55.080759 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:55.080972 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:55.081034 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:55.081269 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:55.081391 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:55.081467 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:55.081531 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 05:55:55.081556 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260926 05:55:55.086541 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:55.083869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43222 (local address 127.24.255.60:45971)
0926 05:55:55.083969 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:55.083971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:55.083980 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:55:55.084063 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:55.084063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:55.084099 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:55.084161 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:55.084366 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:55.084948 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:55.085771 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:55.085986 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:55.086048 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:55.086263 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:55.086376 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:55.086435 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:55.086451 (+ 16us) server_negotiation.cc:299] Negotiation successful
0926 05:55:55.086472 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":45,"threads_started":1}
I20260926 05:55:55.091287 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:55.088255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.24.255.61:38641)
0926 05:55:55.088401 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:55.088403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:55.088524 (+ 121us) server_negotiation.cc:407] Connection header received
0926 05:55:55.088562 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:55.088563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:55.088573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:55.088625 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:55.088717 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:55.089483 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:55.090439 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:55.090704 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:55.090765 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:55.090892 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:55.090927 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:55.090970 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:55.091183 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 05:55:55.091213 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260926 05:55:55.091490 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:55.088492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.24.255.60:45971)
0926 05:55:55.088533 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:55.088534 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:55.088774 (+ 240us) server_negotiation.cc:407] Connection header received
0926 05:55:55.088835 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:55.088836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:55.088847 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:55.088893 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:55.088994 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:55.089460 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:55.090585 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:55.090839 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:55.090891 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:55.091064 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:55.091100 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:55.091151 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:55.091383 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 05:55:55.091408 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:55:55.091663 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:55:55.088309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.24.255.62:33937)
0926 05:55:55.088451 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 05:55:55.088452 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:55:55.088460 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:55:55.088506 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:55:55.088507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:55:55.088561 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:55:55.088647 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:55:55.088813 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:55.089557 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:55.090633 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:55:55.090910 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:55:55.090971 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:55:55.091156 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:55:55.091253 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:55:55.091300 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:55:55.091538 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 05:55:55.091566 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":51,"threads_started":1}
I20260926 05:55:55.098865 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 15755
I20260926 05:55:55.103927 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 15830
I20260926 05:55:55.108690 25596 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGUnhcE/build/release/bin/kudu with pid 15905
2026-09-26T05:55:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2751 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5548 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (273944 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST